Upload
others
View
13
Download
0
Embed Size (px)
Citation preview
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 1
Unified Fabric Maurizio Portolani
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2
• Introduction
• Spanning Tree and vPC
• Fabric Extender
• vPC and FEX design best practice
• FabricPath
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3
Scale-up and Scale-out a Key Requirement
TOP CAREABOUTS IMPACT FABRIC
General Purpose
High Frequency Trading
and High Performance
Computing
Cloud and Service Provider
Web 2.0 and Big Data
• Large Scale L2/L3 Fabric
• Multi-tenancy
• Simplified Management
• Security
• Convergence
• L2/L3 Fabric Scale
• Operational Continuity
• Simplified Management
• Server Virtualization
• Convergence
• Multicast at Scale
• Low Latency
• East/West traffic
• L3 Fabric at Scale (10,000+)
• Open API
• Auto Provisioning and Monitoring
Need For Architectural Flexibility
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
CONVERGENCE
SCALE
INTELLIGENCE
Foundational Technologies
DCB/FCoE
VDC
FEX
Architecture
FabricPath
vPC
OTV
LISP
Consolidated I/O
Virtualizes the Switch
Simplified Management
with Scale
Architectural Flexibility
Active-Active Uplinks
Workload Mobility
Scalability and Mobility
Deployment Flexibility Unified Ports
IO Accelerator Replication and Back up
SME/DMM Compliance and Workload Mobility
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
• Wire once and walk away
• Lower complexity
• Does not disrupt existing Infrastructure
• Eliminate parallel networks – no stranded assets, less space, less heat, less cooling
• Simpler cabling – easier installation, better airflow
• Interoperates with existing LAN and SAN
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
Virtualising the Network Port
LAN LAN
Switch port extended over
Fabric Extender
Lo
gic
al S
wit
ch
Switch
Switch
Legacy multi-tier architecture FEX architecture
Switch
FEX
Collapse network tiers, fewer network management points
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Where Is the Edge?
NIC
PCI-E Bus
Operating System and
Device Drivers
FC 3/11
HBA
Edge of the Network and Fabric
Eth 2/12
pNIC
PCI-E Bus
Hypervisor provides virtualization of PCI-E
resources
FC 3/11
HBA
Eth 2/12
Edge of the
Fabric
VMF
S
SCSI
VNIC
VETH
Still 2 PCI Addresses
on the BUS
PCI-E Bus
Hypervisor provides virtualization of PCI-
E resources
Edge of the
Fabric
VMFS
SCSI
PCIe
Eth
ern
et
Fib
re C
han
nel
10G
bE
10G
bE
Link
Eth 2/12
vFC 3
Converged Network Adapter provides
virtualization of the
physical Media
VNIC
VETH
PCI-E Bus
Hypervisor provides
virtualization of PCI-E resources
Edge of the Fabric
V
M
FS
S
C
SI
veth 1
vFC 4
802.1BR
Eth
1
FC
2
Eth
3
FC
4
Eth
126
vFC 2
vFC 3
vFC 126
P
as
s
T
hr
u
VNIC
VETH
SR-IOV adapter
provides multiple
PCIe resources
Compute and Fabric Edge are Merging
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Some classic Layer 2 Limitations
• Local problems have network-wide impact, troubleshooting is difficult
• Tree topology provides limited bandwidth
• Tree topology introduces sub-optimal paths
• Each Network device is another management point
A B
STP convergence is disruptive
MAC address tables don’t scale
Host flooding impacts the whole network
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
Fabric simplification
With FabricPath:
• Traffic is routed inside the fabric local problems have local impact
• L2 can be extended without risk
FabricPath
VLAN X
VLAN Y
VLAN Z
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
Larger Distributed Topologies
Server, Storage, Application and Facilities are driving Layer 2 Scalability requirements
Server Virtualization and Clustering driving the need for every / any VLAN everywhere based design
Facilities requirements defining the network topology
• VM requirements along with Data Storage growth mandating a need for more efficient and pervasive network based storage
Technology changes will impact any cabling plant design
Migration to 10GE as the default LoM technology
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 11
L2 Domain Elasticity: vPC, FabricPath/TRILL
OTV LAN extensions
OTV
Device Virtualization: VDCs,
VRF enhancements
MPLS VPN
Location of compute resources is transparent to the user
VN-link
notifications
VM-awareness: VN-link
Port Profiles
OTV
OTV
OTV
IP Mobility: LISP
Multi-tenancy/segmentation: Segment-ID, VXLAN, FabricPath and OTV
Virtualized Workload Mobility Connecting Virtualized Data Centers
OTV
OTV
OTV
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
Evolution of Enterprise Environments
L3
L2
L3
L2
L3 Cloud
From STP ----------------------------------------------- to FabricPath
From 2-way redundancy ----------------------------- to N-way redundancy
From Limited L2 paths ------------------------------- to L2 ECMP
From Classic learning --------------------------------- to conversational learning
From rigid topologies --------------------------------- to highly flexible topologies
From hard pod boundaries -------------------------- to segment overlay extensions
From disparate fabrics ------------------------------- to converged fabric
From virtualization unaware fabric ----------------- to virtualization aware fabric
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
Evolution of SPDC Environments
From Restricted VLAN/L2 Scale ----------- to highly scalable VLAN/MAC Space
From Classic L4-7 Scale/Mobility ---------- Physical/Virtual Scale and Service Mobility
From VLAN-Based Multi-tenancy ---------- Tenant ID Based environments
From GE nodes ------------------------------ to 10G nodes pushing 40G uplink adoption
From limited Hypervisors offerings ------- “open” primarily and other as needed
From Stitch-together Orchestration ------ Integrated Process Flow Driven Orchestration
L3 Cloud
L3
L3
L2
L3 L3 Cloud
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
• Introduction
• Spanning Tree and vPC
• Fabric Extender
• vPC and FEX design best practice
• FabricPath
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 15
! Enable vpc on the switch
dc11-5020-1(config)# feature vpc
! Check the feature status
dc11-5020-1(config)# show feature | include vpc
vpc 1 enabled
• vPC allows a single device to use a port channel across two neighbour switches (vPC peers)
• Eliminate STP blocked ports
• Layer 2 port channel only
• Provide fast convergence upon link/device failure
• Available on Nexus 3000, 5000/5500 and 7000*
Virtual Port Channel - vPC Multi-chassis Etherchannel (MCEC)
MCEC
vPC Peers
MCEC
vPC Peers
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16
L
2
L
3
Core
Aggregati
on
Access
WAN
Core FC
Co
re
Edg
e
LAN and SAN utilize different High Availability Models
SAN is dual fabric, LAN is fully meshed fabric
vPC enables ‘both’ architectures at the edge (single device models not acceptable to SAN customers)
Why vPC and not VSS or Stackwise Co-existence of LAN and SAN
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 17
• Do not disable STP !!
• Configure aggregation vPC peers as root and secondary root
If vPC peer-switch is implemented, both vPC peers will behave a single STP root
• Align STP primary root, HSRP active router and PIM DR with vPC primary peer
• BA is enabled by default on vPC peer-link
• Do not enable Loopguard and BA on vPC (disabled by default)
• Enable STP port type “edge” and port type “edge trunk” on host ports
• Enable STP BPDU-guard globally
• Disable STP channel-misconfig guard if supported by access switches
Spanning Tree Recommendations STP Best Practices
BPDU-guard BPDU-guard
Port Type
Edge / Edge Trunk
routing peer agg1b agg1a
Acc1 Acc2
Disable STP channel-
misconfig guard
vPC primary
VLAN 1-4094 root
MST 0 – 3 root
HSRP Active
PIM DR
vPC primary STP
VLAN 1-4094 sec root
MST 0 – 3 sec root
HSRP Standby
vPC_PL
vPC_PKL
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 18
vPC maintains layer 2 synchronization between peers ‘not’ Layer 3
Don’t use L2 port channel to attach routers to a vPC domain
If both routed and bridged traffic is required:
Individual L3 links for routed traffic
L2 port-channel (vPC) for bridged traffic
vPC Interaction with Layer 3 Do not create an L3 peer over a vPC interface
ECMP LInks
L3
L2
ECMP LInks
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 19
• HSRP active process communicates the active MAC to its neighbour
• Only the HSRP active process responds to ARP requests
• HSRP active MAC is populated into the L3 hardware forwarding tables, creating a local forwarding capability on the HSRP standby device
• No need to configure aggressive FHRP hello timers as both switches are active
‘peer-gateway’ command allows a vPC peer to respond both the the HSRP virtual and the real MAC address of both itself and it’s peer
dca-n7k1-vdc2(config-vpc-domain)# peer-gateway
Note:
------:: Disable IP redirects on all interface-vlans of this vPC domain
for correct operation of this feature ::------
vPC Interaction with Layer 3 FHRP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20
N5k(config-vpc-domain)# delay restore ?
<1-3600> Delay in bringing up vPC links (in seconds)
N5k(config-vpc-domain)# delay restore 360 Delay activating vPC interfaces
vPC interaction with Routing convergence on system restart
After a vPC device reloads and comes back up routing protocol needs time to reconverge
vPCs may blackhole routed traffic from access to core until layer 3 connectivity is reestablished
vPC Delay Restore
On system restart delays bringing up the vPC interfaces for ‘n’ seconds
Like HSRP preempt delay it needs to be tuned for your network
vPC and Layer 3 InteractionsvPC Delay Restore
vPC
Primary
vPC
Secondary
L3
L2
OSPF
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 21
HSRP Active
HSRP Standby
L2
L3
RMAC-A RMAC-B
N5k-1 N5k-2
Some devices, such as NetApp filer, could send packets to interface MAC rather than the virtual MAC of FHRP protocol
Such packets will be forwarded over peer-link
The “peer-gateway” will address this issue
L3-N5548-1(config-vpc-domain)# peer-gateway
vPC and Layer 3 InteractionsPeer-Gateway
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 22
Parent Switch vPC Peer-link (10 GE Only) VPC Interfaces
N7K-M132XP-12
✓ ✓
N7K-M132XP-12L
✓ ✓
N7K-M108X2-12L
✓ ✓
N7K-F132XP-15
✓ ✓
F2 – 48 x 10G
✓ ✓
M2 40G & 100G
Future Future
For Your Reference
vPC Supported Hardware Nexus 7000
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
vPC Supported Hardware VDC Combinations with F1/F2/M1 Module
F2
Only
VDC
M1-F1
Mix VDC
F2
Only
VDC
M1-F1-F2
Mix VDC
vPC Peer Link
F1 Only VDC – F1 ports used for the peer-link
F2 Only VDC – F2 ports used for the peer-link
M1/F1 Mixed VDC – F1 ports (16K MAC addresses)
M1/F1 Mixed VDC – M1 ports (128K MAC addresses)
✗
F1
Only
VDC
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 24
Switch vPC Peer Link vPC Interfaces
Nexus 5020
✓ ✓
Nexus 5010
✓ ✓
Nexus
5548P/UP
✓ ✓
Nexus
5596UP
✓ ✓
For Your Reference
vPC Supported Hardware Nexus 5000/5500
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 25
vPC – Virtual Port Channel Nexus 3000
vPC is supported on the Nexus 3000 as of
5.0(3)U2(1)
The maximum number of vPC configurable on
the Cisco Nexus 3000 Series Switches is 64
For Your Reference
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26
• Introduction
• Spanning Tree and vPC
• Fabric Extender
• vPC and FEX design best practice
• FabricPath
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
Distributed High
Density Edge
Switching System
(up to 4096 virtual
Ethernet interfaces)
+
Cisco Nexus® 2000 FEX
Cisco Nexus® 5500
Cisco Nexus® 2000 FEX
Cisco Nexus® 7000
+
Cisco FEX-link: Virtualized Access Switch Nexus 2000 Fabric Extender
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 28
• De-Coupling of the Layer 1 and Layer 2 Topologies
• Simplified Management Model, plug and play provisioning, centralized configuration
• Line Card Portability (N2K supported with Multiple Parent Switches – N5K, 6100, N7K)
• Unified access for any server (100M1GE10GE FCoE): Scalable Ethernet, HPC, unified fabric or virtualization deployment
Virtualized
Switch
. . .
Nexus 2000: Virtualized Access Switch Changing the device paradigm
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 29
Parent Switch FEX Supported
# FEX FEX HIF Capabilities
Nexus 5020
N2K-C2148T, N2K-C2248TP
N2K-C2248TP-E, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM,
B22-HP
12 STP Edge Ports
FCoE VF ports
Nexus 5010
N2K-C2148T, N2K-C2248TP
N2K-C2248TP-E, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM,
B22-HP
12 STP Edge Ports
FCoE VF ports
Nexus 5548P/UP
N2K-C2148T, N2K-C2248TP
N2K-C2248TP-E, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM,
B22-HP
24 – L2
8 – L3
STP Edge Ports
FCoE VF ports
FabricPath Edge Ports
CTS Edge Ports
Nexus 5596UP
N2K-C2148T, N2K-C2248TP
N2K-C2248TP-E, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM,
B22-HP
24 – L2
8 – L3
STP Edge Ports
FCoE VF ports
FabricPath Edge Ports
CTS Edge Ports
For Your Reference
Cisco Nexus 5000 Parent Switch Supported Nexus 2000
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 30
Parent Switch FEX Supported # FEX FEX HIF Capabilities
N7K-M132XP-12
N2K-C2248TP, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM* 32 - L2
32 - L3
STP Edge Ports
CTS Edge Ports
N7K-M132XP-12L
N2K-C2248TP, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM* 32 - L2
32 - L3
STP Edge Ports
CTS Edge Ports
N7K-M108X2-12L
N.A. (FEX not Supported)
N.A. N.A. (FEX not Supported)
N7K-F132XP-15
N.A. (FEX not Supported) N.A. N.A. (FEX not Supported)
F2 – 48 x 10G
N2K-C2248TP, N2K-C2224TP,
N2K-C2232PP, N2K-C2232TM* 32 - L2
32 - L3
STP Edge Ports, FabricPath Edge Ports
CTS Edge Ports, FCoE VF ports (CY12)
M2 40G & 100G
Future N.A. Future
For Your Reference
Cisco Nexus 7000 Parent Switch Supported Nexus 2000
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
N2224TP 24 Port 100/1000M Host
Interfaces
2 x 10G Uplinks
N2232TM 32 Port 1/10GBASE-T Host
Interfaces
8 x 10G Uplinks (Module)
N2148T 48 Port 1000M Host
Interfaces
4 x 10G Uplinks
N2248TP 48 Port 100/1000M Host
Interfaces
4 x 10G Uplinks
N2232PP 32 Port 1/10G FCoE Host
Interfaces
8 x 10G Uplinks
FET-10G Cost Effective Fabric
Extender Transceiver
N2248TP-E 48 Port 100/1000M Host
Interfaces
4 x 10G Uplinks
32MB Shared Buffer
B22HP 16 x 1/10G Host Interfaces
8 x 10G Uplinks
Cisco Nexus 2000Series Platform Overview
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32
Virtualized Access Switch Fabric Extender Transceiver (FET)
• Cost-effective transceiver to interconnect Nexus 2000 and Nexus 5000 and 7000 parent switch (only supported on FEX Fabric interfaces)
• SFP+ form-factor
• Multimode fiber (MMF)
• FET with OM3 MMF can operate up to 100m
• FET with OM2 MMF can operate up to 20m
• FET with 62.5/125um MMF can operate up to 10m
• Approximately 1 watt (W) per transceiver
• Incompatible with SR optics
FET supported
only on Fabric
Interfaces
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33
Bridges that support
Interface
Virtualization ports
must support VNTag
and the VIC protocol
NIV uplink ports
must connect to
an NIV capable
bridge or an NIV
Downlink
Hypervisor
NIV downlink
ports may be
connected to an
NIV uplink port,
bridge or NIC
NIV may be
cascaded
extending the
port extension
one additional
level
NIV downlink ports
are assigned a virtual
identifier (VIF) that
corresponds to a
virtual interface on
the bridge and is
used to forward
frames through NIV’s
LIF
VIF NIV capable
adapters may
extending the
port extension
VIF
The FEXLink Architecture provides the ability to extend the bridge (switch) interface to downstream devices
FEXLink associates the Logical Interface (LIF) to a Virtual Interface (VIF)
LIF
Nexus 2000 Fabric Extender FEX Link Architecture
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 34
2248TP-E
N5596-L3-2(config-fex)# hardware N2248TPE queue-limit 4000000 rx
N5596-L3-2(config-fex)# hardware N2248TPE queue-limit 4000000 tx
N5596-L3-2(config)#interface e110/1/1
N5596-L3-2(config-if)# hardware N2348TP queue-limit 4096000 tx
Optimized for BigData, distributed storage, distributed computing, market data, video editing (bursty applications)
Each HIF has 128KB dedicated buffer and each NIF has 64KB dedicated buffer. The remaining 26MB is all shared
Interface allocates buffer from its dedicated pool first and then get more buffer if needed from shared pool until queue limit is reached
Queue limit specify how many buffer can be allocated for each queue.
Tune ingress queue limit when the FEX uplink experience temporary congestion.
Tune egress queue limit for better burst absorption or scenario with many to one traffic pattern
Per FEX buffer tuning
Nexus 2248TP-E Better buffering and counters
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
Extends FEX connectivity into the partner blade chassis
Cisco Nexus 5000 Switch is a single management point for all the blade chassis I/O modules
End-to-end FCoE support
66% decrease in blade management points*
Blade & Rack networking consistency
Interoperable with Nexus 2000 Fabric Extenders in the same Nexus parent switch
HP provides tier 1-2 support. Additional support via Nexus 5000
Cisco Nexus
B22 Series
Blade FEX
* Assuming 10 HP Blade
Chassis, Includes OA Modules
10G Nexus Fabric Extender
[16x 1GE/10GE KR server facing ports]
[8x10GE SFP+ network facing ports]
Over subscription 2:1 to 16:1 depending on number of uplinks
Cisco Nexus B22 Series Fabric Extender FEX Connectivity for the Blade Server Ecosystem
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
3+3 C19 cords
Eight
interconnect
bays usable
for B22 FEX
– 10/1G &
FCoE
Hot-plug
redundant
fans
Network and serial
connectors
Enclosure links
Redundant
Onboard
Administrators
Cisco Nexus B22 Series Fabric Extender FEX Connectivity for HPC C7000 & C3000 Chassis
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
• Introduction
• Spanning Tree and vPC
• Fabric Extender
• vPC and FEX design best practice
• FabricPath
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Redundancy model – Dual Switch with redundant fabric
Provides isolation for Storage topologies (SAN ‘A’ and ‘B’)
Port Channel and Pinning supported for Fabric Link
Redundancy model – Single switch with dual ‘supervisor’ for fabric, data control & management planes
No SAN ‘A’ and ‘B’ isolation
FEX-link & vPC Virtualized Access Switch Nexus 5000/5500 Topologies prior to 5.1(3)N1
vPC Supported with up to 2 x 8
links
Local Etherchannel with up to 8
links
FCoE Adapters supported on 10G
N2K interfaces
Straight Through Dual Homed
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 39
Redundancy model – Dual Switch (each switch supports redundant supervisors)
vPC Supported with NX-OS 5.2
Fabric links supported on N7K-M132XP-12, N7K-M132XP-12L & N7K-F248XP-25
Port Channel only supported for Fabric Links
FEX-link & vPC Virtualized Access Switch Nexus 7000 Topologies supported as of 5.2
vPC Supported with up to 2 x 8
links
Local Etherchannel with up to 8
links
Nexus 7000 – vPC Nexus 7000 – vPC
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 40
• How is this achieved?
• Configuration associates FCoE traffic to a specific fabric link
switchA(config)# fex 101
switchA(config-fex)# fcoe
switchB(config)# fex 101
CNA
Nexus 2000
Fabric
Extender (FEX)
FCoE
Nexus 5000
(San B) Nexus 5000
(San A)
FCoE
Storage
SAN B SAN A
Enhanced vPC Isolating SAN A and SAN B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
• Introduction
• Spanning Tree and vPC
• Fabric Extender
• vPC and FEX design best practice
• FabricPath
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
“FabricPath brings Layer 3
routing benefits to flexible Layer 2
bridged Ethernet networks”
Easy
Configuration
Plug & Play
Provisioning
Flexibility
Multi-pathing
(ECMP)
Fast
Convergence
Highly Scalable
Switching Routing
FabricPath
Cisco FabricPath NX-OS Innovation Enhancing L2 with L3
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 43
CE Edge
Ports
FP Core
Ports
Spine
Switch
Leaf
Switch
Interface connected to traditional network device
Sends/receives traffic in standard 802.3 Ethernet frame format
Participates in STP domain
Forwarding based on MAC table
Classical Ethernet (CE)
S
10
S
20
S
30
S
40
S10
0
S20
0
S300
1/
1 1/
2
FabricPath (FP)
A B
Interface connected to another FabricPath device
Sends/receives traffic with FabricPath header
Does not run spanning tree
Does not perform MAC learning!
Exchanges topology info through L2 ISIS adjacency
Forwarding based on ‘Switch ID Table’
Cisco FabricPath Terminology
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
Classical Ethernet
(CE)
S
10
S
20
S
30
S
40
S10
0
S20
0 S30
0
1/
1
• The association MAC address/Switch ID is maintained at the edge
• Traffic is encapsulated across the Fabric S300: CE MAC
Address Table
MAC IF
B 1/2
… …
MAC IF
B 1/2
A S100
1/
2
S300:
FabricPath
Routing Table
Switc
h
IF
… …
S100 L1, L2, L3,
L4
FabricPath (FP)
Switch ID
space:
Routing
decisions are
made based
on the
FabricPath
routing table
MAC adress
space:
Switching
based on MAC
address tables
S100 S300 A
B
A B
Cisco FabricPath A New Data Plane
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 45
S100: CE MAC
Address Table
A
S
10
S
20
S
30
S
40
S10
0
S20
0 S300
FabricPath
B
1/
1
Classical
Ethernet
S300: CE MAC
Address Table
MAC IF
B 1/2
… …
S200: CE MAC
Address Table
MAC IF
… …
… …
S100 M A
B
Lookup B: Miss
Don’t learn
Lookup B: Miss
Flood
Lookup B: Hit
Learn source A
MAC IF
B 1/2
A S100
MAC IF
… …
… …
MAC IF
A 1/1
… …
1/
2
FabricPath Key Concept #1 Conversational MAC Learning
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 46
Classical
Ethernet Conversational
Learning
S100: CE MAC
Address Table
A
S
10
S
20
S
30
S
40
S10
0
S20
0
S300
FabricPath
B
1/
1 S300: CE MAC
Address Table
MAC IF
B 1/2
… …
S200: CE MAC
Address Table
MAC IF
… …
… …
MAC IF
B 1/2
A S100
MAC IF
… …
… …
MAC IF
A 1/1
… …
1/
2
S300:
FabricPath
Routing Table
Switc
h
IF
… …
S100 L1, L2, L3,
L4
S300 S100 B A
Lookup A: Hit
Send to S100 Lookup A: Hit
Learn source B
MAC IF
A 1/1
B S300
FabricPath Key Concept #1 Conversational MAC Learning
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 47
S1
00
S
10
S
20
S
30
S
40
S2
00
S3
00
FabricPath
• Describes shortest (best) paths to each Switch ID based on link metrics
• Equal-cost paths supported between FabricPath switches
FabricPath
Routing Table on
S100 Switc
h
IF
S10 L1
S20 L2
S30 L3
S40 L4
S200 L1, L2, L3,
L4
… …
S300 L1, L2, L3,
L4
One ‘best’ path
to S10 (via L1)
Four equal-cost
paths to S300
FabricPath Key Concept #2 It’s a Routed Network
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 48
S
10
S
20
S
30
S
40
• Multi-destination traffic constrained to loop-free trees touching all FabricPath switches
• Root switch elected for each multi-destination tree in the FabricPath domain
• Loop-free tree built from each Root assigned a network-wide identifier (Ftag)
• Support for multiple multi-destination trees provides multipathing for multi-destination traffic
Two multi-destination trees supported in NX-OS release 5.1
Root for
Tree 1
S100 S
20
Root for
Tree 2
S
10 S200
S300
S
30
S
40 Logical
Tree 1
Root
S
40
S100
S200
S300
S
10
S
20
S
30 Logical
Tree 2
Root
S1
00 S2
00
S3
00
FabricPath
FabricPath Key Concept #3 Multicast
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 49
FabricPath
MAC Table on
S100 MAC IF/SID MAC IF/SID
A e1/13 (local)
e2/29
S
10
S
20
S
30
S
40
Root for
Tree 1
Root for
Tree 2
S1
00 S2
00
S3
00
MAC A MAC B
Multidestinatio
n
Trees on
Switch 100 Tre
e
IF
1 po10
2 po10,po20,po30
,po40
Broadcast →
DMAC→FF
SMAC→A
Payload
Multidestinatio
n
Trees on
Switch 10 Tre
e
IF
1 po100,po200,po
300
2 po100
po10 po20
po40
po30
Ftag →
Ftag →
DMAC→FF
SMAC→A
Payload
DA→FF Ftag→1
SA→100.0.12
DMAC→FF
SMAC→A
Payload
po100
po300
po200
e1/13
po10
po20 po30 po40
1
3
2
4
6
DMAC→FF
SMAC→A
Payload
DA→FF Ftag→1
SA→100.0.12
Learn MACs of directly-connected
devices unconditionally
Don’t learn MACs from
flood frames
FabricPath
MAC Table on
S200 MAC IF/SID
Multidestinatio
n
Trees on
Switch 300 Tre
e
IF
1 po10,po20,po30
,po40
2 po40
5
Putting it all together – Host A to Host B (1) Broadcast ARP Request
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 50
e2/29
S
10
S
20
S
30
S
40
Root for
Tree 1
Root for
Tree 2
S2
00
S3
00
MAC A MAC B
po10 po20
po40
po30
po100
po300
po200
e1/13
po10
po20 po30 po40
Multidestinatio
n
Trees on
Switch 100 Tre
e
IF
1 po10
2 po10,po20,po30
,po40
Ftag →
Multidestinatio
n
Trees on
Switch 10 Tre
e
IF
1 po100,po200,po
300
2 po100
Ftag →
11
10
DMAC→A
SMAC→B
Payload
DA→MC1 Ftag→1
SA→300.0.64
DMAC→A
SMAC→B
Payload
7 Unknown →
A →
DMAC→A
SMAC→B
Payload
DA→MC1 Ftag→1
SA→300.0.64
FabricPath
MAC Table on
S300 MAC IF/SID MAC IF/SID
B e2/29 (local)
8
MISS
If DMAC is known, then
learn remote MAC
Multidestinatio
n
Trees on
Switch 300 Tre
e
IF
1 po10,po20,po30
,po40
2 po40
9 FabricPath
MAC Table on
S100 MAC IF/SID
A e1/13 (local)
MAC IF/SID
A e1/13 (local)
B 300.0.64
(remote)
12 DMAC→A
SMAC→B
Payload
*MC1 = 01:0f:ff:c1:01:c0
Putting it all together – Host A to Host B (2) Broadcast ARP Reply
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 51
e2/29
S
10
S
20
S
30
S
40
S2
00
S3
00
MAC A MAC B
po10 po20
po40
po30
e1/13
po10
po20 po30 po40
S1
00
DMAC→B
SMAC→A
Payload
FabricPath
MAC Table on
S100
DMAC→B
SMAC→A
Payload
13 MA
C IF/SID
A e1/13 (local)
B 300.0.64
(remote)
B →
14
S300 →
FabricPath
Routing Table
on S100 Switc
h
IF
S10 po10
S20 po20
S30 po30
S40 po40
S200
po10,
po20,
po30,
po40
S300
po10,
po20,
po30,
po40
DMAC→B
SMAC→A
Payload
DA→300.0.64 Ftag→1
SA→100.0.12
15
S300 →
FabricPath
Routing Table
on S30 Switc
h
IF
… …
S300 po300
16
S300 →
Hash
DMAC→B
SMAC→A
Payload
DA→300.0.64 Ftag→1
SA→100.0.12
FabricPath
MAC Table on
S300 MA
C IF/SID
B e2/29 (local)
MA
C IF/SID
A S100.0.12
(remote)
B e2/29 (local)
18
FabricPath
Routing Table
on S300 Switc
h
IF
… …
S300 Use LID
(64)
17
If DMAC is known, then
learn remote MAC
po300
Putting it all together – Host A to Host B Unicast Data - Routed
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 52
Switch CE edge port FP core port Routing for
FP VLAN
Nexus 5020
✗ ✗ ✗
Nexus 5010
✗ ✗ ✗
Nexus 5548P/UP
*
✓ ✓ ✓
Nexus 5596UP *
✓ ✓ ✓
For Your Reference
* L3 daughter card is ‘not’ needed in 5550 to run
FabricPath, only L3 routing for FabricPath VLANs
FabricPath Hardware & Software Nexus 5500
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 53
I/O Module CE edge port FP core port Routing for FP
VLAN
N7K-M132XP-12
✗ ✗ ✓
N7K-M148GT-11
✗ ✗ ✓
N7K-M148GS-11
N7K-M148GS-11L ✗ ✗ ✓
N7K-M108X2-12L
✗ ✗ ✓
N7K-F132XP-15 ✓ ✓ ✗
N7K-F248XP-25 * ✓ ✓ ✓
* F2 module needs to be in its own VDC or system. It is not possible to mix F1 and
F2 LC in the same VDC
For Your Reference
FabricPath Hardware & Software Nexus 7000
Cisco Confidential © 2010 Cisco and/or its affiliates. All rights reserved. 54
The Nexus 7000 features two kinds of IO Modules:
M series and F series.
M I/O Modules cannot switch FabricPath traffic
When running FabricPath,
FP Core and CE Edge ports must be on an F module or 5500
New FabricPath/CE locally significant VLAN mode:
FabricPath VLANs can only be enabled on F modules (FEX+F2 with NX-OS 6.0
release as well) or 5500 (5500 + FEX as well)
S100(config)# vlan 10
S100(config-vlan)# mode ?
ce Classical Ethernet VLAN mode
fabricpath Fabricpath VLAN mode
S100(config-vlan)# mode fabricpath
S100(config-vlan)#
FabricPath
E
F FabricPath Core
Port
Classical
Ethernet Edge
Port
FabricPath Design – VLAN Mode F1/F2/M1 Interaction
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 55
SVI SVI
F1 M1
VLAN 10 = CE VLAN
switchport VLAN 10
switchport VLAN 10
F1 M1
VLAN 10 = CE VLAN VLAN 20 = FP VLAN
switchport VLAN 10
switchport VLAN 20
M1
VLAN 10 = FP VLAN
switchport VLAN 10
F1
VLAN 10 = CE VLAN VLAN 20 = FP VLAN
switchport trunk VLAN 10,20 CE FP
M1
VLAN 10 = CE VLAN VLAN 20 = FP VLAN
switchport trunk VLAN 10,20
CE
F1
VLAN 10 = CE VLAN VLAN 20 = FP VLAN
switchport trunk VLAN 10,20
FP
Bridging in CE VLAN from M1 port to F1 port in CE mode
Routing via SVIs from M1 port in CE VLAN to F1 port in FP VLAN
F1 trunk port in CE mode carrying CE VLAN and FP VLAN
F1 port in FP mode carrying CE VLAN and FP VLAN
M1 trunk port carrying CE VLAN and FP VLAN
M1 port in FP VLAN
Note : F2 needs to be on
its own VDC so the FP
and CE VLAN
compatibility mode does
not apply
FabricPath Design F1/F2/M1 Interaction
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 56
• Introduction
• Spanning Tree and vPC
• Fabric Extender
• vPC and FEX design best practice
• FabricPath
• Summary
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 57
Leading with Innovation
NX-OS
Nexus 7K
• Industry leading scale and density
• Enabling scalable fabrics: FabricPath
• High-Availability with hitless ISSU
• Workload Mobility: OTV & LISP
• DC Consolidation: VDCs, FCoE
Nexus 2K
• Remote linecard for N5K & N7K
• 1GE & 10GE optimized options with FCoE
• Consistent architecture for blade and rack servers
Nexus 5K
• Low latency Non-blocking
• Unified ports: 1/10GE, FCoE, 2/4/8G FC
• Enabling Scalable Fabrics: FabricPath
• Virtualization: Adapter FEX, VM-FEX
Blade Offerings
• Nexus 4K : 10GE FCoE blade switch for IBM
• B22 HP: Fabric extender for HP blade server chassis
Nexus 1K
• Visibility and security for virtual machines
• Support for multiple hypervisors
• Services integration
Nexus 3K
• Ultra low latency 1/10/40GE Switching
• Optimized for high-performance workloads
• Rich L2/L3 features
Thank you.