Upload
marius-diaconu
View
224
Download
0
Embed Size (px)
Citation preview
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
1/97
2006 Cisco. Al l r ights reserved. Cisco Conf ident ialPresentation_ID 1
Virtualization inData CenterUnified Fabric with Nexus
Maciej Bocian
Architecture Sales Manager
Data Center and Virtualization, Central Europe
CCIE#7785
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
2/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 2
Traditional Data Center Network Topology
L2
VLAN A
Module 1
VLAN B
L3
VLAN D VLAN EVLAN C
L3
L2
Core
Aggregation
Access
Hierarchical Design
Triangle and Square Topologies
Multiple Access Models: Modular, Blade Switches and ToR
Multiple Oversubscription Targets (Per Application Characteristics)
2000 10000 Servers
10,000 to 50,000 ports
Module 2
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
3/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 3
New Data Center Architecture
Topology Layers:Core Layer: Support high density L3 10GE aggregation
Aggregation Layer: Support high density L2/L3 10GE aggregation
Access Layer: Support EoR/MoR, ToR, & Blade for 1GE, 10GE, DCE & FCoE attached servers
Topology Service:Services through service switches attached at L2/L3 boundary
Topology Flexibility:Pod-wide VLANs, Aggregation-wide VLANs or DC-wide VLANs
Trade off between flexibility and fault domain
Agg-wide VLANs
DC-wide VLANs
Pod-wide VLANs
L2
L3
L2L3
SAN Fabric
Fabric AFabric B
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
4/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 4
Physical facilities,
cabling andstandards
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
5/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 5
Data Center Strategy in ActionPhysical Facilities
Servers
Network
Pod
4 - 6 Zones Per DC & 6 15 MW per DC
60,000 80,000 SQF per zone 1-3 MW per zone200 400 racks/cabinets per zone
Cooling and power per pod (per pair of rack rows)
8 48 servers per rack/cabinet 1-1.5 KW percabinet
2 11 interfaces per server
2500 30000 server per DC
4000 120,000 ports per DC
Zone
DC
Pod
Storage
It all depends on server typesand network access layer model
COLD AISLE
HOT AISLE
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
6/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 6
Reference Physical TopologyNetwork Equipment and Zones
Server Rack
Network Rack
Zone
DC
Pod
Storage Rack
COLD AISLE
HOT AISLE
Pod
Pod
Module 1 Module N
Pod
Pod
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
7/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 7
Pod ConceptNetwork Zones and Pods
Pod
Pod/Module SizingTypically mapped to access topology Size: determined by distance and density Cabling distance from server racks to network racks
100m Copper 200-500m Fiber
Cable density: # of servers by I/Os per serverRacks Server: 6-30 Servers per rack Network (based on access model) Storage: special cabinets
DC SizingDC: a group of zones (or clusters, or areas)Zone: Typically mapped to aggregation pairNot all use hot-cold aisle designPredetermined cable/power/cooling capacity
DC
COL D AISLE
HOT AISLE
Pod
Pod
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
8/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 8
Network Equipment DistributionEnd of Row and Middle of Row
Patch panel Patch panel
NetworkAccess Point
A - B
End of Row
server
server
server
server
Patch panelX-connect
NetworkAccess Point
C - D
Patch panelX-connect
Patch panel
NetworkAccess Point
A - B
Middle of Row
server
server
serverPatch panelX-connect
NetworkAccess Point
C - D
Patch panelX-connect
Patch panel
End of Row
Traditionally usedCopper from server to access switchesPoses challenges on highly dense server farms
Distance from farthest rack to access point Row length may not lend itself well toswitch port density
Middle of RowUse is starting to increase given EoR challengesCopper from servers to access switchesIt addresses aggregation requirements for ToRaccess environmentsFiber may be used to aggregate ToR
Common CharacteristicsTypically used for modular accessCabling is done at DC build-outModel evolving from EoR to MoR
Lower cabling distances (lower cost)Allows denser access (better flexibility)
6-12 multi-RU servers per Rack4-6 Kw per server rack, 10Kw-20Kw per networkrackSubnets and VLANs: one or many per switch.Subnets tend to be medium and large: /24, /23
server
Fiber
Copper
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
9/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 9
Network Equipment DistributionTop of Rack
Top of Rack
server
server
Top of Rack
server
ToRUsed in conjunction with dense accessracks(1U servers)Typically one access switch per rack
Some customers are considering two +cluster
Use of either side of rack is gaining traction Cabling:
Within rack: Copper from server toaccess switchOutside rack (uplink):
Copper (GE): needs a MoR model for fiberaggregationFiber (GE or 10GE):is more flexible and alsorequires aggregation model (MoR)
Subnets and VLANS: one or many subnets per access switch Subnets tent to be small: /24, /25, /26
Patch panel
Network
AggregationPointA - B
server
server
serverPatch panelX-connect
Network
AggregationPointA - B
Patch panelX-connect
Patch panel
server
Top of Rack Top of Rack
NetworkAggregation
PointA - B
Patch panelX-connect
NetworkAggregation
PointC - D
Patch panelX-connect
Top of Rack
server
Top of Rack
Patch panel Patch panel
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
10/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 10
Network Equipment DistributionBlade Chassis
Switch to SwitchPotentially higher oversubscriptionScales well for blade server racks(~3 blade chassis per rack)Most current uplinks are copper butthe newer switches offer fiberMigration from GE to 10GE uplinksis taking place
Pass-throughScales well for pass-through bladeracksCopper from servers to accessswitches
ToRHave not seen it used inconjunction with blade switchesMay be a viable option on pass-through environments is theaccess port count is rightEfficient when used with BladeVirtual Switch environments
Blade Chassis
sw1 sw2
Blade Chassis
sw1 sw2
Blade Chassis
sw1 sw2
Blade Chassis
Pass-through
Blade Chassis
Pass-through
Blade Chassis
Pass-through
NetworkAggregation
PointA B C - D
Patch panelX-connect
NetworkAggregation
PointA B - C - D
Patch panelX-connect
Top of Rack
Blade Chassis
Pass-through
Blade Chassis
Pass-through
Blade Chassis
Pass-through
Blade Chassis
sw1 sw2
Blade Chassis
sw1 sw2
Blade Chassis
sw1 sw2
Network
AggregationPoint
A B C - D
Patch panelX-connect
Network
AggregationPoint
A B - C - D
Patch panelX-connect
Patch panel
Patch panel
Patch panel
Patch panel
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
11/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 11
10 Gigabit Ethernet Server Connectivity
CableTransceiver
Latency (link)Power
(each side)DistanceConnector
(Media)
Twinax ~ 0.1s~ 0.1W
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
12/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 12
Cost Effective 10GE ConnectivityToday
SFP+ USR Ultra Short Reach
100M on OM3 fiber, 30M on
OM2 fiber
Support on all Cisco Catalystand Nexus switches
Low Cost: $995 NTE
SFP+ Direct Attach
1, 3, 5 and 7M on Twinax
0.1W Power
Support across all NexusSwitches
Low Cost: $150 - $260
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
13/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 13
Nexus 7000
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
14/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 14
Nexus 7010 10-Slot Chassis
First chassis in Nexus 7000 product
family Optimized for data center environments
High density256 10G interfaces per system
384 1G interfaces per systems
High performance64 non-blocking 10G ports1.2Tbps system bandwidth at initial
release
80Gbps per slot
60Mpps per slot
Future proofInitial fabric provides up to 4.1TbpsProduct family scaleable to 15+Tbps
40/100G and Unified Fabric ready
33.1-38(84-96.5cm)
17.3 (43.9cm)
21 RU36.5
(92.7cm)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
15/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 15
Nexus 7010 Chassis
Optional frontdoors
Front Rear
System statusLEDs
Integrated cablemanagement
with cover
Supervisorslots (5-6)
Payload slots(1-4, 7-10)
Air intake withoptional filter
Air exhaust
Crossbar fabric
modules
System fan trays
Power supplies
Fabric fan trays
21RU
ID LEDs onall FRUs
Front-to-back airflow
Lockingejectorlevers
Common equipmentremoves from rear
Two chassisper 7 rack
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
16/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 16
Nexus 7018 18-Slot Chassis
Second chassis in Nexus 7000 product family
Ultra-high density512 10G interfaces per system
768 1G interfaces per system
High performance
128 non-blocking 10G ports
2.5Tbps system bandwidth at initial release
80Gbps per slot
60Mpps per slot
Future proof
Initial fabric provides up to 7.8Tbps
Chassis scaleable to 17.6Tbps
40/100G and Unified Fabric ready
Shared equipment
Supervisors, I/O modules, power suppliescommon between chassis
Fabrics and fan trays chassis-specific33.1-38
(84-96.5cm)
17.3 (43.9cm)
25 RU43.5
(110.5cm)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
17/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 17
Nexus 7018 Chassis
Optional frontdoors
Front Rear
System statusLEDs
Integrated cable
management
Supervisorslots (9-10)
Power supplyair intake
Crossbarfabric
modules
Power supplies
25RU
ID LEDs onall FRUs
Side-to-sideairflowLocking
ejector
levers
Common equipmentremoves from rear
Systemfan trays
Payload slots(1-8, 11-18)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
18/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 18
Nexus 7000SupervisorEngine
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
19/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID19
Compact Flash cover
Supervisor Engine
Dual-core 1.66GHz Intel Xeon processor with 4GB DRAM Connectivity Management Processor (CMP) for lights-out
management
2MB NVRAM, 2GB internal bootdisk, 2 external compact flash slots
10/100/1000 management port with 802.1AE LinkSec
Console & Auxiliary serial ports USB ports for file transfer
Blue beacon LED for easy identification
BeaconLED
Console Port
AUX PortManagement
Ethernet
USB Ports CMP Ethernet
Reset ButtonStatusLEDs
Compact Flash
Slots
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
20/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 20
Out-of-bandmanagement
network
CMP
CMP
CMP
CMP
DataNetwork
CMP
CMP
CMP
CMP
Connectivity Management Processor (CMP) Standalone, always-on microprocessor on
supervisor engine
Provides lights out remote management anddisaster recovery via 10/100/1000 interface
Removes need for terminal servers
Monitor supervisor and modules, access log
files, power cycle supervisor, etc.Runs lightweight Linux kernel and network stack
Completely independent of DC-OS on main CPU
DataNetwork
console cab les
Terminal Servers(out-of-band
console connectivity)
Out-of-band
managementnetwork
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
21/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 21
Nexus 7000I/O Modules
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
22/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 22
32-Port 10GE I/O Module
32 10GE ports with SFP+transceivers
80G full duplex fabric connectivity
Integrated 60Mpps forwardingengine for fully distributedforwarding
4:1 oversubscription at front panel
Virtual output queueing (VOQ)ensuring fair access to fabricbandwidth
802.1AE LinkSec on every port
Buffering:
Dedicated mode: 100MB ingress,80MB egress
Shared mode: 1MB + 100MBingress, 80MB egress
Queues: 8q2t ingress, 1p7q4tegress
Blue beacon LED for easyidentification
SFP+
SR at initial release 300m over MMFLR post-release 10km over SMF
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
23/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 23
32-Port 10GE I/O Module Architecture
2,4 6,8 10,12 14,16 18,20 22,24 26,28 30,32
Fabric Interface
and VOQLayer 2Engine
Fabric Interface
and VOQ
Fabric ASIC
To Fabr ics
Port ASIC Port ASIC Port ASIC Port ASIC
CTS and4:1 Mux
CTS and4:1 Mux
CTS and4:1 Mux
CTS and4:1 Mux
MAC/PHY
CTS and4:1 Mux
CTS and4:1 Mux
CTS and4:1 Mux
CTS and4:1 Mux
Port ASIC Port ASIC Port ASIC Port ASIC
Mezzanine Card
1,3 5,7 9,11 13,15 17,19 21,23 25,27 29,31
Layer 3Engine
FE Daugh te r
Card
LCCPU
To Central ArbiterEOBC
(to Por t ASIC)(to L C CPU)
Inband
ReplicationEngine
MET
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
MAC/PHY
ReplicationEngine
MET
ReplicationEngine
METReplication
EngineMET
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
24/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 24
48-Port 1GE I/O Module
48 1GE 10/100/1000 RJ-45 ports 40G full duplex fabric connectivity
Integrated 60Mpps forwardingengine for fully distributedforwarding
Virtual output queueing (VOQ)ensuring fair access to fabricbandwidth
802.1AE LinkSec on every port Buffer: 7.5MB ingress, 6.2MB
egress
Queues: 2q4t ingress, 1p3q4tegress
Blue beacon LED for easyidentification
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
25/97 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 25
48-Port 1GE I/O Module Architecture
ReplicationEngine
MET
Port ASIC
17-24
Octal PHY
Layer 2Engine
Fabric ASIC
FE Daugh te r
Card
To Fabr ics
Layer 3
Engine
Fabric Interfaceand VOQ
CTS CTS CTS
Port ASIC
CTS CTS CTS
Port ASIC
CTS CTS CTS
Port ASIC
CTS CTS CTS
9-16
Octal PHY
1-8
Octal PHY
41-48
Octal PHY
33-40
Octal PHY
25-32
Octal PHY
To Central Arbiter
LCCPU
EOBC
(to Por t ASIC)
(to L C CPU)
Inband
ReplicationEngine
MET
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
26/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 26
Nexus 7000ForwardingEngine
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
27/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 27
Forwarding Engine HardwareAdvanced hardware forwarding engine integrated on every I/O module
60Mpps Layer 2 bridging with hardware MAC learning
60Mpps IPv4 and 30Mpps IPv6 unicast
IPv4 and IPv6 multicast (SM, SSM, bidir)
IPv4 and IPv6 security ACLs
Cisco TrustSec security group tag support
Unicast RPF check and IP source guard QoS remarking and policing policies
Ingress and egress NetFlow (full and sampled)
GRE tunnels
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
28/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 28
Forwarding Engine Details
Forwarding engine chipset consists of two ASICs:
Layer 2 EnginePerforms ingress and egress SMAC/DMAC lookups
Hardware MAC learning
True IP-based Layer 2 multicast constraint
Performs lookups on ingress I/O module, and egress I/O module for bridgedpackets
Layer 3 Engine60Mpps IPv4 and 30Mpps IPv6 Layer 3/Layer 4 lookups
Performs all FIB, ACL, QoS, NetFlow processing
Linear, pipelined architecture every packet processed in ingress and egress
pipePerforms lookups on ingress I/O module, and egress I/O module for multicastreplicated packets
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
29/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 29
Nexus 7000Fabric andBandwidth
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
30/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 30
Fabric Module
Provides 46Gbps per I/Omodule slot
Also provides 23G persupervisor slot
Up to 230Gbps per slot with 5
fabric modulesInitially shipping I/O modules
do not leverage full fabricbandwidth
Load-sharing across all fabricmodules in chassis
Multilevel redundancy withgraceful performance degradation
Non-disruptive OIR
Blue beacon LED for easyidentification
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
31/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 31
46Gbps92Gbps138Gbps184Gbps230Gbps
Fabric Capacity and Redundancy Per-slot bandwidth capacity increases with each fabric module
1G module requires 2 fabrics for N+1 redundancy
10G module requires 3 fabrics for N+1 redundancy
4th and 5th fabric modules provide additional level of redundancy
Future modules will leverage additional fabric bandwidth
Fabric failure results in reduction of overall system bandwidth
Fabrics
ModuleSlots
40G
1G Module
80G
10G Module
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
32/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 32
Access to Fabric Bandwidth
Supervisor engine controls access to fabric bandwidthusing central arbitration
Fabric bandwidth represented byVirtual Output Queues
(VOQs)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
33/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 33
Virtual Output Queues (VOQs) on ingress modules representbandwidth capacity on egress modules
Guaranteed delivery to egress module for arbitrated packetsentering fabric
If VOQ available on ingress, capacity exists on egress
VOQ is NOT equivalent to ingress or egress port buffer or queuesRelates ONLY to ASICs at ingress and egress to fabric
VOQ is virtual because it represents EGRESS capacity but resideson INGRESS module
It is PHYSICAL buffer where packets are stored
What Are VOQs?
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
34/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 34
What Is VOQ?Ingress module
Module 1 Module 2(1G module)
Module 3(10G module)
Module 4(10G module)
VOQs forModule 2
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
VOQs forModule 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
VOQs forModule 4
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
Egress modules
Fabricmodule
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
Destination 1
Destination 2
Destination 3
Destination 4
Destination 5
Destination 6
Destination 7
Destination 8
Destination 1
Destination 2
Destination 3
Destination 4
Destination 5Destination 6
Destination 7
Destination 8
0 1 2 30 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
Destination 1
Destination 2
Destination 3
Destination 40 1 2 3
0 1 2 3
0 1 2 3
0 1 2 3
EgressCapacity
(ability to receive trafficfrom fabric)
VOQ Buffers correspondto Egress Capacity
(send traffic into fabric basedon destination)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
35/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 35
Centralized Fabric Arbitration
Access to fabric bandwidth on ingress module controlledby central arbiter on supervisor
In other words, access to the VOQ for the destination across the fabric
Arbitration works on credit request/grant basis
Modules communicate egress fabric buffer availability to central arbiterModules request credits from supervisor to place packets in VOQ fortransmission to destination over fabric
Supervisor grants credits based on egress fabric buffer availability forthat destination
Arbiter discriminates among four classes of servicePriority traffic takes precedence over best-effort traffic across fabric
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
36/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 36
CentralArbiter
Module 2
Fabrics
VOQ Operation
Supervisor
Buffer
Credits
VOQ for
e2/1,3,5,7
VOQ for
e1/1,3,5,7
0 1 2 3VOQ for
e3/1,3,5,7
0 1 2 3 0 1 2 3
Capacity
available!
Capacity
available! Capacity
available!
Module 1 Module 3
0 1 2 3Egress
Destination
Capacity
Egress
Destination
Capacity
0 1 2 3Egress
Destination
Capacity
0 1 2 3
Egress modules havecapacity to receive t raffic
from fabric
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
37/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 37
Fabrics
VOQ Operation
Supervisor
Module 1 Module 2 Module 3
0 1 2 3VOQ for
e3/1
0 1 2 3Egress
Destination
Capacity
0 1 2 3Egress
Destination
Capacity
0 1 2 3VOQ for
e2/1
INGRESS MODULE EGRESS MODULES
VOQs on ingress modulecorrespond to capacity
on egress modules
CentralArbiter
Buffer
Credits
VOQ for
e2/1,3,5,7
VOQ for
e1/1,3,5,7
0 1 2 3VOQ for
e3/1,3,5,7
0 1 2 3 0 1 2 3
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
38/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 38
Fabrics
VOQ Operation
Supervisor
Module 1 Module 2 Module 3
0 1 2 3VOQ for
e3/1 Destined toe3/1, priorit y
level 1
Request to
transmit to
e3/1, prior ity 1!
Request
granted!
0 1 2 3Egress
Destination
Capacity
Buffer for VOQ
prior ity 1 now
available!
0 1 2 3Egress
Destination
Capacity
0 1 2 3VOQ for
e2/1
INGRESS MODULE EGRESS MODULES
CentralArbiter
Buffer
Credits
VOQ for
e2/1,3,5,7
VOQ for
e1/1,3,5,7
0 1 2 3VOQ for
e3/1,3,5,7
0 1 2 3 0 1 2 3
Deduct cred it
fr om VOQ
priority 1
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
39/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 39
Benefits of Central Arbitration andVOQ
Ensures fair access to bandwidth for multiple ingressports transmitting to one egress port
Prevents congested egress ports from blocking ingresstraffic destined to other ports
Priority traffic takes precedence over best-effort trafficacross fabric
Engineered to support Unified I/O
Can provide no-drop service across fabric for future FCoEinterfaces
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
40/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 40
Layer 2Engine
Layer 3Engine
Forwarding Engine
Nexus 7000 Packet Flow
Fabric Module 1
Fabric ASIC
Fabric Interfaceand VOQ
Port ASIC
CTS and4:1 Mux
MAC/PHY
MAC/PHY
ReplicationEngine
Fabric ASIC
Module 1e1/1
Layer 2Engine
Layer 3Engine
Forwarding Engine
Fabric Interfaceand VOQ
Port ASIC
CTS and4:1 Mux
MAC/PHY
MAC/PHY
ReplicationEngine
Fabric ASIC
Module 2
Supervisor Engine
Central Arbiter
Fabric Module 2
Fabric ASIC
Fabric Module 3
Fabric ASIC
e2/7
Ingressqueueing andscheduling
CTS LinkSec decryption andverification
Ingress queueing and schedulingin shared mode
Submit packetfor lookup
Ingressmulticast
replication Layer 2 andIGMP snooping
lookups
Layer 3
and Layer4 lookups
Queueing andVOQ arbitrationrequest
Transmit tofabric
Credit grant forfabric access
Packet transmission
Packet transmission
Receive fromfabric
Queue andschedule
toward egress Return buffer
credits
Submit packetfor lookup
Egressmulticast
replication
Layer 2 andIGMP snooping
lookups
Layer 3and Layer4 lookups
Egressqueueing andscheduling
CTSLinkSec
encryption
Receivepacket
fromwire
Transmitpacket on
wire
Packet transmission
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
41/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 41
Virtual DeviceContexts
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
42/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 42
Virtual Device Contexts (VDCs)
VDC Virtual DeviceContext
Flexible separation/distribution ofSoftware Components
Flexible separation/distribution ofHardware Resources
Securely delineatedAdministrative Contexts
Infrastructure
Layer-2 Protocols Layer-3 Protocols
VLAN mgr
STP
OSPF
BGP
EIGRP
GLBP
HSRP
VRRP
UDLD
CDP
802.1XIGMP sn.
LACP PIMCTS SNMP
RIBRIB
Protocol Stack (IPv4 / IPv6 / L2)
Layer-2 Protocols Layer-3 Protocols
VLAN mgr
STP
OSPF
BGP
EIGRP
GLBP
HSRP
VRRP
UDLD
CDP
802.1XIGMP sn.
LACP PIMCTS SNMP
RIBRIB
Protocol Stack (IPv4 / IPv6 / L2)
Kernel
VDC
VDC B
VDC A VDC B
VDC n
VDCs are notThe ability to run different OS levelson the same box at the same time
based on a hypervisor model; thereis a single infrastructure layer that
handles h/w programming
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
43/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 43
Virtual Device ContextsAn Introduction to the VDC Architecture
Virtual Device Contexts provides virtualization at the device level allowing multiple instances of the
device to operate on the same physical switch at the same time
Linux 2.6 Kernel
Infrastructure
Protocol Stack (IPv4/IPv6/L2)
L2 Protocols
VDC1
VLAN Mgr
Physical Switch
VDCnProtocol Stack (IPv4/IPv6/L2)
L3 Protocols
UDLD
VLAN Mgr UDLD
LACP CTS
IGMP 802.1x
RIB
OSPF GLBP
BGP HSRP
EIGRP VRRP
PIM SNMP
RIB
L2 Protocols
VLAN Mgr
L3 Protocols
UDLD
VLAN Mgr UDLD
LACP CTS
IGMP 802.1x
RIB
OSPF GLBP
BGP HSRP
EIGRP VRRP
PIM SNMP
RIB
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
44/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 44
Virtual Device ContextsThe Default VDC
When the system is activated for the first time, it will have a default VDC enabled - this VDC is present
at all times during the operation of the switch
Linux 2.6 Kernel
Infrastructure
Protocol Stack (IPv4/IPv6/L2)
L2 Protocols
VDC1
VLAN Mgr
Physical Switch
L3 Protocols
UDLD
VLAN Mgr UDLD
LACP CTS
IGMP 802.1x
RIB
OSPF GLBP
BGP HSRP
EIGRP VRRP
PIM SNMP
RIB
Default VDC VDC #1 (DEFALT_VDC) is the default VDC - bydefault, all ports in the physical chassis are
assigned to the default VDC while not assigned to
any other VDC
Users cannot create or delete the default VDC
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
45/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 45
Virtual Device ContextsVDC Fault Domain
A VDC builds a fault domain around all running processes within that VDC - should a fault occur in a
running process, it is truly isolated from other running processes and they will not be impacted
Linux 2.6 Kernel
Infrastructure
Protocol StackVDCA
Physical Switch
VDC A
ProcessAB
C
ProcessDE
F
ProcessXY
Z
Protocol StackVDCB
VDC B
ProcessAB
C
ProcessDE
F
ProcessXY
Z
Fault Domain
Process DEF in VDC Bcrashes
Processes in VDC A arenot affected and will
continue to run unimpeded
This is a function of theprocess modularity of theOS and a VDC specific
IPC context
Nexus 7000 Roadmap
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
46/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 46
Nexus 7000 RoadmapLow er ing P ow er Cos t w h i le I n c r eas ing S ca le
15+ TerabitInfrastructure
M1 Series I/O Modules32 por t 10G (80G/slot)48 por t 1G (46G/slot)
M1 Series I/O Module48 port 1G (46G/slot)
M1 Series I/O Modu les8 por t 10G-XL (80G/slot )48 port 1G-XL (46G/slot)
M1 Series I/O Modules16 port 10G (160G/slot )
D1 Series I/O Modules32 port 10G DCB SFP+ (230G/slot )
32 por t 10G DCB 10GBASE-T (230G/slot)D2 Series I/O Modules
48 port 10G DCB SFP+ w/L3 (480G/slot)48 port 10G DCB 10GBASE-T w/L3 (480G/slot)
40G/100G mod
vPC1GbE
18 Slot Chassis
OTV, MPLSN2K Suppor t
DCB , L2MP, FCoE
LISPCCN
Service Modules
ISSUCTSVDC
2008 2010 20112009
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
47/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 47
Nexus 5000
Nexus 2000
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
48/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 48
OS
Cisco Nexus 5000 Series
56-Port L2 Switch
40 Ports 10GE/FCoE/DCE, fixed
2 Expansion module slots
Cisco Fabric Manager and Cisco Data Center Manager
Cisco DC-OS
FC + Ethernet
4 Ports 10GE/FCoE/DCE
4 Ports 1/2/4G FC
Fibre Channel
8 Ports 1/2/4G FC
Mgmt
Cisco DC-OS
Ethernet
6 Ports 10GE/FCoE/DCE
DC-NM and Fabric Manager
NX-OS
28-Port L2 Switch
20 Ports 10GE/FCoE/DCE, fixed
1 Expansion module slot
Nexus 5010Nexus 5020
All 10GE switch/module ports are FCoE/Data Center Ethernet capable
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
49/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 49
Nexus 2000 Fabric Extender1GE Connectivity
48 x 1 GE interfaces 4 x 10 GE interfaces
Beacon and status LEDs
Redundant, hot-swappable
power supplies
Hot-swappable fan tray
N 2000 F b i E t d
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
50/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 50
Nexus 2000 Fabric ExtenderVirtual Chassis
The Nexus 2000 Fabric Extender (FEX) acts as a remote linecardfor the Nexus 5000, retaining all centralized management and configurationon the Nexus 5000, transforming it into a Virtualized Chassis
Nexus 5000Virtualized chassis
+
Nexus 5000
Nexus 2000 Fabric Extender=
D t C t A A hit t
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
51/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 51
Data Center Access ArchitectureVirtualized Access Switch
Nexus 5010/5020
Nexus 5000/2148T Virtualized
Access Switch provides a number ofdesign options to address evolvingData Center requirements
Fabric Extender provides for flexibilityin the design of the physicaltopologies
Aids in building larger layer 2 designssafely
Support of latest spanning treeenhancements
Single virtual access switch(Simplifies the layer 2 design)
Support of 16-way 10GEEtherchannel combined with vPCprovides increased network capacity
Nexus 2148T FabricExtender 48 GE Ports
4 x 10GE FabricLinks per FabricExtender (CX-1
Cu)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
52/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 52
Data Center ArchitectureN5K/N2K - Logical Topology
Nexus 5000/2000Vir tualized Access
Switch Pods . . .
Cisco Nexus 2148T FabricExtender (N2K) and Nexus5000 (N5K) Pod
N2K + N5K Podrepresents networking
Access layer Nexus 7000 at Distribution
Layer
Each VirtualizedAccess Switch Podconfigured to supportup to 576 1GE server
ports at FCS
D t C t A A hit t
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
53/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 53
Data Center Access ArchitectureOptimizing Layer 1 and Layer 2 Designs
1GE Attached Servers - Maintain Existing Cat5e Server
Wiring Infrastructure with EoR topology
Nexus 5000/2000EoR
. . .
Cisco Nexus 2148T Fabric Extender and Nexus
5000 provide a Flexible Access Solution De-Coupling of the Layer 1 and Layer 2 Topologies
Optimization of both Layer 1 (Cabling) and Layer 2(Spanning Tree) Designs
Provides for simultaneous support of EoR, MoR and
ToR
D t C t A A hit t
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
54/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 54
Data Center Access ArchitectureN5K/N2K Advantages Flexible Cabling
Combination of EoR and ToR cabling
Nexus 5000/2000Mixed ToR & EoR
. . .
Cisco Nexus Fabric Extender (FEX) and Nexus
5000 provide a Flexible Access Solution
Migration to ToR for 10GE servers or selective1GE server racks if required (mix of ToR and EoR)
Mixed cabling environment (optimized as required)
Flexible support for Future Requirements
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
55/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 55
Fabric ExtenderFabric Modes
Fabric Extender associates (pins)a server side (1GE) port with anuplink (10GE) port
Server ports are either individuallypinned to specific uplinks (staticpinning) or all interfaces pinned to
a single logical port channel Behavior on FEX uplink failure
depends on the configuration
Static Pinning Server portspinned to the specific uplink are
brought down with the failure ofthe pinned uplink
Port Channel Server traffic isshifted to remaining uplinks basedon port channel hash
Static Pinning
Port Channel
Server Interfacegoes down
Server Interfacestays active
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
56/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 56
Nexus 2148 Fabric ExtenderConfiguring the Fabric Extender
Two step process
Define the Fabric Extender (100-199) and the number of fabricuplinks to be used by that FEX (valid range: 1-4)
Nexus- 5000# swi t ch# conf i gur e t ermi nalswi t ch( conf i g) # f ex 100swi t ch( conf i g- f ex) # pi nni ng max- l i nks 4
Nexus- 5000# swi t ch# swi t ch# conf i gur e t er mi nalswi t ch( conf i g) # i nt er f ace et her net 1/ 1swi t ch( conf i g- i f ) # swi t chpor t mode f ex- f abr i cswi t ch( conf i g- i f ) # f ex associ at e 100. . .
Configure Nexus 5000 ports as fabric ports and associate thedesired FEX
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
57/97
D t C t A A hit t
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
58/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 58
Data Center Access ArchitecturevPC Redundancy Models Dual Chassis
MCEC fr om server to theaccess switch
vPC provides two redundancy designs for the virtualized access switch
Option 1 - MCEC connectivity from the server
Two virtualized access switches bundled into a vPC pair
Full redundancy for supervisor, line card, cable or NIC failure
Logically a similar HA model to that currently provided by VSS
vPC peers
Two Virtualized access switchesEach with a Sing le Supervisor
D t C t A A hit t
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
59/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 59
Data Center Access ArchitecturevPC Redundancy Models Dual Supervisor
Act ive/Standby NIC teaming
vPC Option 2 Fabric Extender connected to two Nexus 5000
From the server perspective a single access switch with each line cardsupported by redundant supervisors
Full redundancy for supervisor, fabric via vPC and cable or NIC failurevia active/standby NIC redundancy
Logically a similar HA model to that currently provided by dual
supervisor based modular switch
Fabric Extender dual homed toredundant Nexus 5000
N 5000 & 2000 R d
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
60/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 60
Nexus 501028-Port 1RU Switch
Nexus 502056-Port 2RU Switch
Q2CY08
Q4CY08
Nexus 5000 & 2000 Roadmap
FEX-1GE
N2148T-1GE
48x1GE + 4x10GE
FEX-100M/1GT
N2248T-1GE48 port 100/1GT
downlinks, 4x10GEuplinks
Q1/Q2CY10
Q1CY09
Next-Generation Nexus 500048-ports & 96-port s Switch
FEX-10GE
N2232-10GE32 ports 10GE
SFP+ downlinks,8x10GE SFP+
uplinks
2HCY10
N2K
N5K
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
61/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 61
Nexus 1000V
The Story TodayNetworking with
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
62/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 62
The Story TodayNetworking withVI3.5
Separation of Network andServer provisioning andmanagement systems
Virtual Center managing& provisioning ESX hostsand vSwitches
Physical network
managed andprovisioning separately
Network visibility ends atphysical switch port
Different interfaces and tools
IOS or IOS-like cli forphysical network
VC GUI and esxcfg cli forvSwitches
vSwitch vSwitch vSwitch
NetworkManagement
Virtual Center
vNetwork Distributed Switch & Cisco Nexus
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
63/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 63
vNetwork Distributed Switch & Cisco Nexus1000V
vSwitchCURRENT
vSwitch vSwitch
VDS
vNetwork Distributed Switch Cisco Nexus 1000V
Enterprise networking vendors canprovide proprietary networkinginterfaces to monitor, control andmanage virtual networks
First offering: Cisco Nexus 1000V
Virtual machines retain policies,QoS as they move around thedatacenter
Ci N 1000V C t
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
64/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 64
Cisco Nexus 1000V Components
Cisco VEM
VM1 VM2 VM3 VM4
Cisco VEM
VM5 VM6 VM7 VM7
Cisco VEM
VM9 VM10 VM11 VM12
Virtual Ethernet Module(VEM)
Replaces Vmwares virtual switch
Enables advanced switching capabilityon the hypervisor
Provides each VM with dedicatedswitch ports
vCenter Server
Virtual Supervisor Module(VSM)
CLI interface into the Nexus 1000V
Leverages NX-OS 4.04a
Controls multiple VEMs as a singlenetwork device
Cisco VSMs
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
65/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 65
Cisco Nexus 1000V Virtual Chassis
Cisco VEM
VM1 VM2 VM3 VM4
Cisco VEM
VM5 VM6 VM7 VM8
pod5- vsm# show modul eMod Por t s Modul e- Type Model St atus
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -1 0 Vi r t ual Supervi sor Modul e Nexus1000V act i ve *2 0 Vi r t ual Supervi sor Modul e Nexus1000V ha-s t andby
3 248 Vi r t ual Et hernet Modul e NA ok
Cisco VSMs
Nexus 1000V Faster VM Deployment
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
66/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 66
Nexus 1000V Faster VM Deployment
Nexus 1000V VSMvCenter
vSphere
Nexus1000VVEM
vSphere
Nexus1000VVEM
Defined Policies
WEB AppsHR
DB
DMZ
VM Connection Policy
Defined in the network
Applied in Virtual Center
Linked to VM UUID
Policy-BasedVM Connectiv ity
Mobil ity of Network &Securi ty Propert ies
Non-DisruptiveOperational Model
Cisco VN-Link: Virtual Network Link
VM
VM
VM
VM
VM
VM
VM
VM
Nexus 1000V Richer Network Services
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
67/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 67
Nexus 1000V Richer Network Services
Nexus 1000V VSM
vSphere
Nexus1000VVEM
vSphere
Nexus1000VVEM
VN-Link Property Mobility
VMotion for the network
Ensures VM security
Maintains connection state
VMs Need to Move
VMotion
DRS
SW Upgrade/Patch
Hardware Failure
vCenter
Policy-BasedVM Connectivity
Mobil ity of Network &Securi ty Propert ies
Non-DisruptiveOperational Model
Cisco VN-Link: Virtual Network Link
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
VM
Nexus 1000V Increased Operational Efficiency
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
68/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 68
Nexus 1000V - Increased Operational Efficiency
Nexus 1000V VSM
vSphere
Nexus1000VVEM
vSphere
Nexus1000VVEM
vCenter
Network Admin Benefits Unifies network mgmt and ops
Improves operational security
Enhances VM networkfeatures
Ensures policy persistence
Enables VM-level v isibility
VI Admin Benefits Maintains existing VM mgmt
Reduces deployment time
Improves scalability
Reduces operational workload
Enables VM-level v isibility
Policy-BasedVM Connectiv ity
Mobil ity of Network &Secur ity Propert ies
Non-DisruptiveOperational Model
Cisco VN-Link: Virtual Network Link
VM
VM
VM
VM
VM
VM
VM
VM
Key Features of the Nexus 1000V
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
69/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 69
Key Features of the Nexus 1000V
Switching L2 Switching, 802.1Q Tagging, VLAN Segmentation, Rate Limiting (TX)
IGMP Snooping, QoS Marking (COS & DSCP)
Security Policy Mobility, Private VLANs w/ local PVLAN Enforcement
Access Control Lists (L24 w/ Redirect), Port Security
Provisioning Automated vSwitch Config, Port Profiles, Virtual Center Integration
Optimized NIC Teaming with Virtual Port Channel Host Mode
Visibility VMotion Tracking, ERSPAN, NetFlow v.9 w/ NDE, CDP v.2
VM-Level Interface Statistics
Management Virtual Center VM Provisioning, Cisco Network Provisioning, CiscoWorks
Cisco CLI, Radius, TACACs, Syslog, SNMP (v.1, 2, 3)
Cisco Nexus 1000V
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
70/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 70
Cisco Nexus 1000V3 new fea tu res t ha t m ake a d i ffe r ence
Great for mixed useESX clusters
Segment VMs w/oburning IP addresses
Supports isolated,community and
promiscuous trunkports
Follows your VM w/VMotion or DRS
Private VLANs(PVLANs)
View flow based statsfor individual VMs
Captures multi-tieredapp traffic inside a single
ESX host
Export aggregate statsto dedicated collector for
DC-wide VM view
Follows your VM w/VMotion or DRS
Netflow v.9 withData Export
Mirror VM interfacetraffic to a remote sniffer
Identify root cause forconnectivity issues
No host based sniffervirtual appliance to
maintain Follows your VM w/
VMotion or DRS
EncapsulatedRemote SPAN
(ERSPAN)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
71/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 71
Differnet aspects of
Virtualizationin Data Center
Virtualization in the Data Center
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
72/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 72
Virtualization in the Data Center
Common Benefits Increase resource usability Lower CAPEX Loosely couple re-usable functions - Flexibility Dynamic allocation of virtual instances - Automation Centralized policy Management Lower TCO
Distributed Capabilities Broad use of functions
Servers: the capability of decoupling CPU, Mem and I/O functions fromphysical devices that provide them to increase their effective utilization,
to enhance the flexibility in how they are utilized and to allow thedynamic management of logical instances
Storage: the capability of abstracting the physical location of datastorage by presenting logical storage to the user for thus achievinglocation independence
Network-based Services: the capability of manipulating serviceinstances independent from the service device thus providing flexibilityin their usage
Network Infrastructure: the capability of partitioning groups ofnetworked resources providing logical isolation and common policy
Virtual Ethernet Switching
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
73/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 73
Virtual Switches: Logical instances of physical switches
- Many to one: grouping of multiple physical switches
- Reduce management overhead (single switch) and simplify configuration (single sw config)
- One to Many: partitioning of physical switches
- Isolate control plane and control plane protocols
Virtual PortChannels: Etherchannel across multiple chassis
- Simplify L2 pathing by supporting non-blocking cross-chassis concurrent L2 paths
- Lessen reliance on STP (loopfree L2 paths are not established by STP)
Virtual Switching Implementations
- Virtual Switching System VSS: Catalyst 6500
- Virtual Blade Switches VBS: 10GE-based Blade Switches
- Virtual Device Context VDC: Nexus 7000
- Virtual Port-Channel vPC: Nexus Family
Virtual Ethernet SwitchingImproving Management and Pathing
Virtual Switching
Virtual Switching:
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
74/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 74
Many to OneMany switches look like one
Two switches are physicalOne switch is virtual
Virtual Switch:i. All ports appear to be on the same physical switch
ii. Single point of managementiii. Single configurationiv. Single IP/macv. Single control plane protocol instance
Benefitsi. Simplify infrastructure managementii. 1 switch to manage
A1AA2
A1AA2
STP HSRP
OSPF SNMP
STP HSRP
OSPF SNMPSTP HSRP
OSPF IGMP
Virtual Switching:Many to One VSS
Virtual Blade Switching
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
75/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 75
Many to OneMany switches look like one
Up to Eight switches are physicalOne switch is virtual
Virtual Switch:
i. All ports appear to be on the same physical switchii. Single point of managementiii. Single configurationiv. Single IP/mac
Benefitsi. Simplify infrastructure managementii. 1 switch to manage
AA
A2A1
A8A7
A4A3
A6A5
A2A1
A8A7
A4A3
A6A52
2
Virtual Blade SwitchingMany to One VBS
Virtual Switching
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
76/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 76
One to ManyOne switch looks like one
1 switch is physicalMany switches are logical
Virtual Switch:i. Switch ports only exist on a single logical instance
ii. Per virtual switch point of managementiii. Per virtual switch configurationiv. Per virtual switch IP/macv. Per virtual switch control plane protocol instance
Benefitsi. Control plane isolationii. Control protocol isolation
Virtual SwitchingOne to Many VDC
A
A1 A2
A3 A4
A
A1 A2
A3 A4
STP HSRP
OSPF IGMP
STP HSRP
OSPF IGMP
STP HSRP
OSPF IGMP
STP HSRP
OSPF IGMP
STP HSRP
OSPF IGMP
Virtual Switching
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
77/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 77
Virtual SwitchingOne to Many VDC Topology
AG1 AG2
AC
VLAN X
AC1 AC2
VLAN X
AG11
AG21
AG12
AG22AG1 AG2
AC1 AC2
VLAN Y
AC11 AC12
VLAN XVLAN Y
AC21 AC22
VLAN X
AG12
AG22
AG11
AG21
One to ManyOne switch looks like many
Devices connect to a single virtual switchVirtual Switching Topologies are isolated from one anotherVirtual Topology:
i. Distinct physical ports form virtual topologiesii. Each topology has independent and isolated control protocols
Benefitsi.Support Isolated but parallel topologiesii.Supports smaller logical environments
Virtual Portchannels
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
78/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 78
Two to oneTwo Physical to a single logical
Devices connect to a single logical switchConnections are treated as portchannelVirtual PortChannel:
i. Ports to virtual switch could form a cross-chassis portchannelii. virtual Portchannel behaves like a regular Etherchannel
Benefitsi.Provide non-blocking L2 pathsii.Lessen Reliance on STP
A
AG1 AG2
2 2
4
2 2
AG1 AG2
2 2
H
AG
8
ACAC1 AC2
1 1
2
Virtual PortchannelsTopology VSS and vPC (through MCEC)
H
New Topology Using Virtual Switching
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
79/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 79
New Topology Using Virtual Switching
BA
4 4
4
2 4 2 2
4
Physical View Logical View
Key characteristicsOptimized L2 Pathing through Virtual PortChannels
i. Increase bandwidth usage on: server to switch and switch to switchii. Loop-free forwarding paths
Less Reliance on STPi. Loopfree topology established through Virtual Switch mechanismii. STP is strictly used as a fail-safe mechanism
Isolation of L2 Domainsi. Separate Logical Topologiesii. Distinct STP Topologies
core1 core2
agg2agg1
acc2acc1
agg4agg3
accYaccN
Virtualswitch
Virtualswitch
Virtualswitch
Virtualswitch
Virtualswitch
C D BAC D
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
80/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 80
New standards:
Fibre Channel overEthernet
What Is FCoE?
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
81/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 81
What Is FCoE?Fibre Channel over Ethernet
From a Fibre Channel standpoint its
FC connectivity over a new type of cable called an Ethernetcloud
From an Ethernet standpoints itsYet another ULP (Upper Layer Protocol) to be transported, but
a challenging one!
And technically
FCoE is an extension of Fibre Channel
onto a Lossless Ethernet fabri c
FCoE
FC over Ethernet (FCoE)
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
82/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 82
Encapsulate Fibre Channel framesonto Lossless Ethernet
FCoE
FC over Ethernet (FCoE)
FibreChannelTraffic
Ethernet
Destination MAC Address
Source MACAddress
IEEE 802.1Q Tag
ET = FCoE Ver Reserved
Reserved
Reserved SOF
Encapsulated FC Frame(Including FC-CRC)
EOF Reserved
FCS
Reserved
FCoE Frame Format
Bit 0 Bit 31
Ethernet
Header
FCoE
Header
FC
Header
FC Payload CRC
EOF
FCS
Byte 0 Byte 2179
Fibre Channel over Ethernet
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
83/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 83
Fibre Channel over EthernetBrief look at the Technology
A method for a direct mapping of FC framesover Ethernet
Seamlessly connects to FC networks
Extends FC in the datacenter over the Ethernet
FCoE appears as FC to the host and the SAN
Preserves current FC infrastructure
and managementFC frame is unchanged
Can operate over standard switches(with jumbo frames)
Priority Flow Control guarantees no-drops
Mimics FC credit-buffer system, avoidsTCP
Does not require expensive off-loads
FibreChannelTraffic
Ethernet
FCoE
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
84/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 84
FC HBA
FC HBA
NIC
NIC
FC Traffic
FC Traffic
Enet Traffic
Enet Traffic
FC HBA
FC HBA
NIC
NIC
FC HBA
FC HBA
NIC
NIC
Today: Parallel LAN/SAN Infrastructure
Inefficient use of NetworkInfrastructure
5+ connections per server higheradapter and cabling costs
Adds downstream port costs;cap-ex and op-ex
Each connection adds additionalpoints of failure in the fabric
Power and cooling
Longer lead time for serverprovisioning
Multiple fault domains complexdiagnostics
Management complexity firmware,driver-patching, versioning
Unified I/O Use Case
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
85/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 85
Management
SANB
SANA
LAN
Today:
Ethernet
FC
Aggregation/Coreswitches
Access Top of the
Rack switches Servers
Unified I/O Use Case
FC HBA
FC HBA
NIC
NIC
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
86/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 86
Management
SANB
SANA
LAN
FCoE
Ethernet
FC
Today
Unified I/O Use Case
Unif ied I/O Phase 1
FCoESwitch
Unified I/O Reduction of server adapters
FewerCables
Simplificationof access layer & cabling
Gateway free implementation - fits ininstalled base of existing LAN and SAN
L2 Multipathing Access Distribution
Lower TCO
Investment Protection (LANs and SANs)
Consistent Operational Model
One set of ToR Switches
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
87/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 87
How the design
has changed
Discrete Network FabricsT i l Eth t d St T l FC E
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
88/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 88
Typical Ethernet and Storage Topology
Fabric A Fabric B
SAN Fabric
Enet
FC
Single Ethernet Network Fabric
Typically 3 tiers
Access Switches are dual-homed
Servers are single or multi-homed
VSAN 2 VSAN 3
A B E FC D
L2
VLAN A VLAN B
L3
VLAN C
L3
L2
Core
Aggregation
Access
Dual Storage Fabrics
Typically 2 tiers
Edge sw itches are dual-homed
Servers are dual-homed to d ifferent fabrics
FCoE
Unified Fabric: DCEFCoE Se e A e FCoE
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
89/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 89
FCoE Server Access
Fabric A Fabric B
SAN Fabric
L2
VLAN A VLAN B
L3
L3
L2
Core
Aggregation
Access
Enet
FC
DCE
A B EDVLAN C VLAN D
CNA
CNA Converged Network Adaptor
FCoE
New TopologyE h d L2 D i
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
90/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 90
Enhanced L2 Design
VLAN A
Module 1
VLAN B
L2
Module 2
L3
VLAN D VLAN EVLAN C
aggx aggx+1
accX accX+1 accY accY+1
Enhanced L2 Topology
3-tier L2 Topology
Nexus at Core and Aggregation Layers
6500 at Aggregation and Services Layers
Topology Highlights
DC-Wide VLANs
Higher Stability of STP environment New STP Features
Lower Oversubscription - if NeededHigher Density 10 GE at Core and Agg Layers
acc1 acc2 accN accN+1
core1 core2
agg2agg1
Enhance L2 TopologyE d t d Vi t l S it hi
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
91/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 91
End to end Virtual Switching
VLAN A
Module 1
VLAN B
L2
Module 2
L3
VLAN D VLAN EVLAN C
accX accX+1
Enhanced L2 Topology
3-tier L2 Topology
Nexus at Core and Aggregation Layers
6500 at Aggregation and Services Layers
Topology Highlights
DC-Wide VLANs
Higher Stability of STP environment New STP Features
Lower Oversubscription - if NeededHigher Density 10 GE at Core and Agg Layers
acc1 acc2
accY
agg1 aggx
core1
1 X
ServerN
accN
New Topology Isolating Collapsed L2 DomainsVi t l D i C t t @ A L
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
92/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 92
Virtual Device Contexts @ Agg Layer
Pods are isolated at aggregation layer
Each Pod runs its own STP instance (instanceper VDC)
Multiple pods could exist in a single VDC
VLANs contained within Agg Module per VDC
L2
L3
VLAN C VDC2
VLAN C VDC1
1 2 agg2agg1
Pods are logically isolated two topologiesEach Pod belong to multiple VDCs
Each VDC topology requires dedicated Ports
VLANs contained within Agg Module per VDC
Higher 10GE Port Density Allows multiple Agg Pairs to be collapsed
Collapsed Agg Pair could still be L2 isolated (different STP instances)
VLAN IDs could be replicated on different VDC shared infrastucture
Module 1
L2
Module 1
L3
VLAN C VLAN C
acc1 acc2 accN accN+1
agg1 agg2 agg3 agg4
acc1 acc2 accN accN+1
L2
Module 1
L3
VLAN C VDC1 VLAN C VDC2
acc1 acc2 accN accN+1
agg1 agg2
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
93/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 93
Brief summary
Q&A
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
94/97
IaaS Service= Compute + Storage + Network
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
95/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 95
= Compute + Storage + NetworkBronze Gold
Bronze CPU RAM SAN
Web1 .5 4 500G
DB1 .5 8 500G
App1 .5 16 500G
Silver
Silver CPU RAM SAN
Web2 1 4 Nx500G
DB2 1 8 Nx500G
App2 1 16 Nx500G
Gold CPU RAM SAN
Web3 2 4 Nx500G
DB3 2 8 Nx500G
App3 2 16 Nx500G
Cisco Unified Data Center of 2010
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
96/97
2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 96
Cisco Unified Data Center of 2010
Unified Compute Nexus 70001/10GE / 10GBaseT
FCoE
N2K / N5K1 GE
Nexus 500010 GEFCoE
Top of Rack Desig nsEnd o f Row
N2K / N7K1 GE
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VMVM
VM
VM
VM
VM
VMVM
CoreL3 boundary to the DC network. Functional point forroute summarization, the injection of default routes
and termination of segmented virtual transportnetworks
AggregationTypical L3/L2 boundary. DC aggregation point for
uplink , storage and DC services offering keyfeatures: VPC, VDC, 10GE density and 1st point of
migration to 40GE and 100GE
1Gb to 10Gb to Unified FabricPrice performance
Next gen N5K (48-96 Universal Port)FCoE on N7K
100/1Gb FEX 10G FCoE + 10GBase-T
VM Visibil ityVM Securit y
Optimized VMotionFast VM Provisioning
High Density VM Deploym ent
Access
Virtual AccessNEXUS 1000v
Virtua l Adapter
Expanded Memory
NEXUS 5000NEXUS 2000
NEXUS 7000vPC
Catalyst6500
ServiceModules
NEXUS 7000
NEXUS1000v
MDS-StorageNEXUS 7000vPC, L2MP
FCoE
8/11/2019 Virtualization in Data Center - Unified Fabric With Nexus
97/97