BRKDCT-2610

7/26/2019 BRKDCT-2610

http://slidepdf.com/reader/full/brkdct-2610 1/91

BRKDCT-2610

Next Generation Data Centre Architec

7/26/2019 BRKDCT-2610


© 2011 Cisco and/or its aff il iates. Al l r ights reserved. Cisco PublicBRKDCT-2610

Reference Sessions

BRKDCT-2023 - Evolution of the Data Centre Access Architec

BRKDCT-2011 - Design and Deployment of Data Centre Interc

BRKVIR-2006 - Deployment of VN-Link with the Nexus 1000v

BRKDCT-2621 - Deploying Cisco Layer 2 Multipathing Techn

BRKDCT-1044 - FCoE for the IP Engineer

7/26/2019 BRKDCT-2610



Agenda

Data Centre Facilities and Network InfrastructureChallenges and Trends

Next Generation Data Centre Technologies

–Virtual Port Channels (vPC)

–Fabricpath –Data Centre Interconnect

– Access Layer

7/26/2019 BRKDCT-2610



Data Centre Facilities and NetworkInfrastructure Challenges and Trends

7/26/2019 BRKDCT-2610

http://slidepdf.com/reader/full/brkdct-2610 5/91© 2011 Cisco and/or its aff il iates. Al l r ights reserved. Cisco PublicBRKDCT-2610

DC Environment Trends

1. Physical Infrastructure

1. Power & Space

2. Cooling and Airflow

3. Cabling

4. Racks and Cabinets

2. Network Infrastructure

1. Access

2. Aggregation

3. Core

4. Services

5. Unified Environments

What are the implications…

Brownfield DCs are aging fast and are hard

Greendfield DCs are carefully planned, 18-

Infrastructure choices affect the network arc

What is happening the next 24 m

Migration from GE to 10GE attached servers

Adoption of 40GE technologies: switch interconne

Increase Adoption of Virtualised Technology

Start of migration to non-STP environments: IS-IS

7/26/2019 BRKDCT-2610


Data Centre Evolution Path

Increase in 10 Gigabit Ethernet port density

Tighter integration between servers and the

network

Network/Server demarcation movinginside of the server

Consolidation Virtualisation Automation Utility

7/26/2019 BRKDCT-2610


The Evolving Data Centre ArchitecturEvolution of the Hierarchical Design Access Layer

Layer 2

Layer 3

Access

The Data Centre Architecture has

been based on the hierarchicalswitching design

Aggregation block contains theaccess and aggregation layers

Dedicated service switchesprovide application loadbalancing, firewall, etc.

Servers connected to 1G ports atthe access layer (both ToR andEoR)

Architecture is based on optimiseddesign for control plane stability andscalability

7/26/2019 BRKDCT-2610


L2 Access“

Plug-and-play

provis ion ing

Practically

“plug-n-play

” – No user configuration is requireforwarding database

It makes it simple to support teaming or L2 multicast for clu

Easy to segment traffic with VLANs

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

Layer 2

Domain

7/26/2019 BRKDCT-2610


Current STP Deployments

Current STP blocks redundant uplinks

VLAN based load balancing

Loop Resolution relies on STP

Primary

Root

S

R

7/26/2019 BRKDCT-2610


Drawbacks of L2 AccessVLAN sprawl

MAC address consumption

BPDU generation is CPU intensive with increasing number of VLA

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

MAC Table

A

VLAN sprawl causes flooding and broadcasts to propagate evenwhere they are not needed

Half of the links in thetopology are blocking

Misconfigurations can causeLayer 2 loops which maymake switchesunmanageable

7/26/2019 BRKDCT-2610



Servers, FCoEattached Storage

L3 Access

Routed Access Topologies alleviate the

consumption of L2 tables No Spanning Tree Recalcuations

All links active and forwarding to

Distribution/Agg

Servers

Smaller subnets to manage andmore L3 configuration points

Difficult migration to Unified wire

topologies

Limited VM Mobility

P R O S

C O N S

7/26/2019 BRKDCT-2610



Evolving Data Centre ArchitectureChallenges for the Classical Design

Hypervisor based server virtualisation andthe associated capabilities (vMotion, …)are changing multiple aspects of the DataCentre design

How large do we need to scale Layer 2?

Where does the storage fabric exist (NAS,

SAN, …) How much capacity does a server need

Where is the policy boundary (security,QoS, WAN acceleration, …)?

Where and how do you connect theservers?

Data Cent

Data Cent

7/26/2019 BRKDCT-2610



Current Challenges in the Virtualised Da

Provisioning of network services for VMs (Port profiles, etc.)

Coordination of VM migration

Lack of visibility of VM to VM traffic

Deployment of advanced functionality down to the VMs (ACLs,

Scaling management applications to match growth in deployed

Lack of common management tools

Difficulty in segregating server and network management funct

7/26/2019 BRKDCT-2610



Next Generation Data Centre Technol

7/26/2019 BRKDCT-2610



The Evolving Data Centre ArchitecturEvolution of the Hierarchical Design Access Layer

Layer 2

Layer 3

Access

The Data Centre Architecture has

been based on the hierarchicalswitching design

Aggregation block contains theaccess and aggregation layers

Dedicated service switchesprovide application loadbalancing, firewall, etc.

Servers connected to 1G ports atthe access layer (both ToR andEoR)

Architecture is based on optimiseddesign for control plane stability andscalability

7/26/2019 BRKDCT-2610



Virtual Port-Channels (vPC)

7/26/2019 BRKDCT-2610



Virtual Port Channel - vPC

vPC is a Port-channeling concept extending link

aggregation to two separate physical switches

Allows the creation of resilient L2 topologies

based on Link Aggregation.

–Eliminates the need for STP in the access-

distribution

Provides increased bandwidth

– All links are actively forwarding

vPC maintains independent control planes

vPC switches are joined together to form a

“domain”

Virtual Port

L2

SiSi SiSi

Increased BW

Non-vPC

Physical Topology

vPC domain

7/26/2019 BRKDCT-2610



Virtual Port Channel – vPC

vPC allows a single device to use aport channel across two neighbourswitches (vPC peers)

Eliminate STP blocked ports

Layer 2 port channel only

Provide fast convergence upon

link/device failure

7/26/2019 BRKDCT-2610



vPC and Spanning-Tree

STP for vPCs is controlled bythe vPC operationally primary switch andonly such device sends out BPDUs on STPdesignated ports.

This happens irrespectively of where thedesignated STP Root is located

The vPC operationally secondary deviceproxies STP BPDU messages from accessswitches toward the primary vPC

Primary

vPC (root)

BP

7/26/2019 BRKDCT-2610



vPC Peer Switch

vPC Peer-link

S1 S2

S3 S4

vPC Primary vPC Secondary

vPC1 vPC2

S5

S1

vPC Primary

S6

Peer-switch

RootRoot Root

BP D U

The two vPC peers send the same information:they look like a single root bridge

vPC Peer-link is a regular STP link; it is always in FWD st

vPC VLAN

Logical representPhysical representation

S3S5

R

Peer-sw

7/26/2019 BRKDCT-2610



Virtual Port Channel - vPCvPC Control Plane - FHRP

vPC maintains dual active control planes and

STP still runs on both switches HSRP active process communicates the

active MAC to its neighbour

Only the HSRP active process responds to ARP requests

HSRP active MAC is populated into the L3hardware forwarding tables, creating a localforwarding capability on the HSRP standbydevice

Consistent behaviour for HSRP, VRRP andGLBP

No need to configure aggressive FHRP hellotimers as both switches are active

HSRP Active

HW Programmed to for

sent to the FHRP MAC

BOTH Switch

7/26/2019 BRKDCT-2610



Layer 3 and vPC DesignsLayer 3 and vPC Design Use L3 links to hook up routers and peer with a vPC domain

Don

’t use L2 port channel to attach routers to a vPC domain unless youroute to HSRP address

If both, routed and bridged traffic is required, use individual L3 links for roand L2 port-channel for bridged traffic

Router

7k1 7k2

Switch

Po1

Po2

L3ECMP

Po

P

P

Routing Protocol Peer

Dynamic Peering

Relationship

P

PP P

7/26/2019 BRKDCT-2610



ASA

NX5K

vPC10

ASA with LACP Support

active

ASA ke

vPC11

vPC40

vPC80

7/26/2019 BRKDCT-2610



FabricPath

7/26/2019 BRKDCT-2610



L2 Requires a Tree

Brancheintercon

Spanning Tree Protocol (STP) typically used to build this tr Tree topology implies:

Wasted bandwidth → increased oversubscription

Sub-optimal paths

Conservative convergence (timer-based)

11 Physical Links 5 Log

S1

S2

S3

7/26/2019 BRKDCT-2610



Existing L2 Technology Is Not Perfect

Even in a vPC topology, the design is less flexible than a r

topology, because it’s not possible to distribute traffic amothan 2 aggregation devices

Loops are still possible; this is not a problem in routed netw

Layer 2 tables are not used efficiently, flooding causes l2 t

populated with unnecessary MAC addresses

Is it possible to bring the advances of L3 into the world of L

Ci F b i P th

7/26/2019 BRKDCT-2610



Cisco FabricPathScaling and Simplifying Layer 2 Ethernet Netw

-All Links Active

Traditional Spanning Tree Based Network

-Blocked Links

Cisco FabricPath Netw

s

Eliminate Spanning tree limitations

Multi-pathing across all links, high cross-sectional bandwidth

High resiliency, faster network re-convergence

Any VLAN, any where in the fabric eliminate VLAN Scoping

7/26/2019 BRKDCT-2610



The Layer 2 Evolution

Spanning-Tree vPC Fabr

POD

Bandwidth

Active Paths

Up to 10 Tbps Up to 20 Tbps Up to 16

Single Dual 16 W

Infrastructure Virtualisation and Capacity

Layer 2 Scalability

7/26/2019 BRKDCT-2610


© 2011 Cisco and/or its aff il iates. Al l r ights reserved. Cisco PublicBRKDCT-2610 Cisco Nexus Platform

Cisco NX-OS

Cisco FabricPath Overview

No MAC learning via flooding

Routing, not bridging

Built-in loop-mitigation

Time-to-Live (TTL)

RPF Check

Data Plane Innovation

Plug-n-Play Layer 2 IS-IS

Support unicast and multica

Fast, efficient, and scalable Equal Cost Multipathing (EC

VLAN and Multicast Pruning

Control Plane Innova

Cisco FabricPath

7/26/2019 BRKDCT-2610



FabricPath Feature Set

16-Way Equal Cost Multipathing (ECMP) at Layer 2

FabricPath Header

Hierarchical addressing with built in loop mitigation (RPF,TTL)

Conversational MAC Learning

–Efficient use of hardware resource by learning only MACs for interestedhosts

Access Swit

Up to

16 Switches

FabricPath

Interoperability with existing classic

Ethernet networks• VPC + allows VPC into a L2 Fabric

• STP Boundary Termination

Multi-Topology – providing traffic

engineering capabilities

7/26/2019 BRKDCT-2610



STP DomainFabricPath

STP Domain 1 STP Domain 2

Data Plane Operation

FabricPath header is imposed by ingress switch

Ingress and egress switch addresses are used to make “Routing” deci

No MAC learning required inside the L2 Fabric

A C

S11 S42

C

A

DATA

C

A

DATA

FabricPath

Header

Ingress Switch

S11

S42

Eg

S11 S42Fabric

A C A C

A C

Encapsulation to creates hierarchical address scheme

7/26/2019 BRKDCT-2610



Control Plane Operation

Assigned switch addresses to all FabricPath enabled switches automa

(no user configuration required) Compute shortest, pair-wise paths

Support equal-cost paths between any FabricPath switch pairs

L1L2

S1 S2 S3 S4

S11 S12L2 Fabric

L3

L4

FabricPath Routing Table

Switch IF

S1 L1

S2 L2

S3 L3

S4 L4

S12 L1, L2, L3, L4

… …

S42 L1, L2, L3, L4

Plug-N-Play L2 IS-IS - used to manage forwarding topology

7/26/2019 BRKDCT-2610



Unicast with FabricPath

Support more than 2 active paths (up to 16) across the Fabric

Increase bi-sectional bandwidth beyond port-channel High availability with N+1 path redundancy

Forwarding decision based on ‘FabricPath Routing Table’

A

L1L2

S1 S2 S3 S4

S11 S12L2 Fabric

L3

L4

Switch IF

… …

S42 L1, L2, L3, L4

MAC IF

A 1/1

… …

C S42

1/1

7/26/2019 BRKDCT-2610



Multicast with FabricPath

Several ‘Trees’ are rooted in key location inside the fabric

All Switches in L2 Fabric share the same view for each ‘Tree’ Multicast traffic load-balanced across these ‘Trees’

Forwarding through distinct ‘Trees’

A

L2 Fabric

Root for

Tree #1

Root for

Tree #2

Ingress switch for

FabricPath decides which

“tree” to be used and addtree number in the header

7/26/2019 BRKDCT-2610



Loop Mitigation with FabricPath

Block redundant paths to ensureloop-free topology

Frames loop indefinitely if STPfailed

Could results in complete networkmelt-down as the result of flooding

Minimise impact of transient loop with TTL and RPF Check

STP Domain

Root

L2 Fabric

S1

S10

TTL=3

TTL=2

TTL=0

TTL is part of FabricPat

Decrement by 1 at each

Frames are discarded w

RPF check for multicas“tree” info

Root

7/26/2019 BRKDCT-2610



VLAN Pruning in L2 Fabric

V L 1 0

V L 2 0

V L 3 0

V L 1 0

V L 3 0

V L 2 0

L2 Fabric

Shared

Broadcast Tree

L2 Fabric

VLAN 10

L2 Fabric

VLAN 20

L2 Fabric

VL

Switches indicate ‘ lo

interested VLANs’ tothe L2 Fabric

Broadcast traffic for a

sent to switches that

requested for it

PC E h f F b i P h

7/26/2019 BRKDCT-2610



vPC+ Enhancement for FabricPath

For Switches at L2 Fabric Edge

vPC is still required to provideactive/active L2 paths for dual-homed CEdevices or clouds

However, MAC Table only allows 1-to-1mapping between MAC and Switch ID

Each vPC domain is represunique ‘Virtual Switch’ to tFabric

Switch ID for such ‘Virtual

used as Source in FabricPa

L2 Fabric

S1 S2

A

B

S3

MAC Table

A ???

MAC Table

B S3

B A Payload

B A PayloadS2S3B A PayloadS1S3

MAC Ta

A S

vPC

L2 Fabric

S1

A

S4

B A PayloadS4S3

vPC+

B A Payload

Mi ti f PC t PC

7/26/2019 BRKDCT-2610



Migration from vPC to vPC+

1. Peer-link & all vPCs must be on F1 ports

2. Add fabricpath virtual switch ID under the VPC domain config on e

(this is disruptive, all VPCs will flap).

3. Configure the VPC+ peer-link as "switchport mode fabricpath".

The vPC+ PL will not learn/synchronise anymore MAC@ across th

4. Previous configuration for vPC (vPC member ports) remain the sa5. Previous configuration for FHRP remain the same

6. Change VLAN from CE mode to FP mode (maybe this would be t

of migration)

C t L3 S i t L2 F b i

7/26/2019 BRKDCT-2610



Connect L3 or Services to L2 Fabric

FabricPath enables multipathing forbridged traffic

However, FHRP allows only 1 activegateway for each host, therefore preventtraffic that needs to be routed to take

advantage of multi-pathing

Layer 3 Network

L3

L2 FHRP

FHRPActive

M

u l t i - p a t h i n g

Provide active/active datFabricPath with no changFHRP

Allow multi-pathing even

Same feature can be lev

service nodes as well

Layer 3 Netwo

L3

L2 FHRP

L2 FabricL2 Fabric

FHRPActive

STP B d T i ti

7/26/2019 BRKDCT-2610



STP Boundary Termination

L2MP Core is presented as a single bridge to all connected CE devices STP BPDUs are processed and terminated by CE Ports

CE devices not interconnected will form separate STP domains

Loops outside L2 Fabric will be blocked within each STP domain

L2 Fabr ic sh ould b e the root for al l connected STP dom ain. CE porblocking state when ‘superior BPDU’ is received

L2 Fabric

ClassicalEthernet

(STP)

FabricPath

(L2 IS-IS)

✖STP

Domain 1STP

Domain 2

F b i P th C fi ti

7/26/2019 BRKDCT-2610



FabricPath Configuration

No L2 IS-IS configuration required

New ‘feature-set’ keyword introduced to allow multiple conditional servrequired by FabricPath to be enabled in one shot

Simplified operational model – only 3 CLIs to get FabricPath up and run

L2 Fabric

N7K(config)# feature-set fabricpath

N7K(config)# vlan 10-19

N7K(config-vlan)# mode fabricpath

N7K(config)# interface e1/1 N7K(config-if)# switchport mode

fabricpath

C ti l MAC L i

7/26/2019 BRKDCT-2610




MAC learning method designed to conserve MAC table entries o

edge switches

–FabricPath core switches do not learn MACs at all

Each forwarding engine distinguishes between two types of MAC

–Local MAC – MAC of host directly connected to forwarding engine

–Remote MAC – MAC of host connected to another forwarding engin

Forwarding engine learns remote MAC only if bidirectional conve

occurring between local and remote MAC

–MAC learning not triggered by flood frames

Conversational learning enabled in all FabricPath VLANs


7/26/2019 BRKDCT-2610



MAC C


FabricPath Core

MAC A

MAC B

FabricPath

MAC Table on S100

MAC IF/SID

A e1/1 (local)

B S200 (remote)

S100

S200

S300

Fab

MAC Ta

MAC

A

B

C

FabricPath

MAC Table on S300

MAC IF/SID

B S200 (remote)

C e7/10 (local)

Transparent Interconnection of Lots o

7/26/2019 BRKDCT-2610



Transparent Interconnection of Lots oLinks (TRILL) and Fabric Path

Fabricpath TRILLFrame routing(ECMP, TTL, RPFC etc…)

Yes Yes

Inter-switch links Point-to-point only Point-to-point OR s

Emulated switch Yes No

FHRP active/active

(AnyCast FHRP in the future)

Yes No

Multiple topologies Yes No

Conversational learning Yes No

FabricPath Summary

7/26/2019 BRKDCT-2610



FabricPath Summary

FabricPath is simple, keeps the attractive aspects of

Transparent to L3 protocolsNo addressing, simple configuration and deployment

FabricPath is scalable

Can extend a bridged domain without extending the risks genera

Layer 2 (frame routing, TTL, RPFC)

FabricPath is efficient

High bi-sectional bandwidth (ECMP)

Optimal path between any two nodes

7/26/2019 BRKDCT-2610



Data Centre Interconnect (DCI)

N

E

Network p

Edge or p

N l

Data Centre Interconnect

7/26/2019 BRKDCT-2610



Long DistanceDC 1

C O R

E

A G G R

A C C E S S

SeServer Cluster

Key Recommendations

vPC Domain id for facing vPC layers should be different

No Bridge Assurance on interconnecting vPCs

BPDU Filter on the edge devices to avoid BPDU propagation

No L3 peering between DCs (i.e. L3 over vPC)

vPC domain 10 vPC domain 20

vPC domain 21vPC domain 11

Rootguard

B

F

BPDUgua

BPDUfilte

- Normal po

R

E

-

-

- -

-

-

-

-

F

F

F

F-

-

- -

-

-

B

N N

N

NN

N

R

-

RRR

RR

ata Ce t e te co ectMulti-layer vPC for Agg and DCI

Data Centre Interconnect

7/26/2019 BRKDCT-2610



Nexus 7010 DC-1 DC-2

Nexus 7010

vPC vPC

Encrypted Interconnect

CTS Manual Mode

(802.1AE 10GE line-rate

encryption)

No ACS is required

Overlay Transport Virtualisation (OTV

7/26/2019 BRKDCT-2610



Overlay Transport Virtualisation (OTV

Ethernet LAN Extension overany Network

Ethernet in IP “MAC routing”

Multi-Data Centre scalability

Simplified Configuration &Operation

Seamless overlay - No networkre-design

Single touch site configuration

High Resiliency

Failure domain isol

Seamless Multi-hom

Maximises availablbandwidth

Automated multi-pa Optimal multicast re

OTV Interface Types

7/26/2019 BRKDCT-2610



OTV Interface Types

Edge Device

Internal Interfaces

External Interface

Overlay Interface

OTV

Internal

Interfaces

L2 L3

Join

Interface

Overlay

Interface

OTV Topology Discussion

7/26/2019 BRKDCT-2610



OTV Topology Discussion

Egress Routing Localisation

7/26/2019 BRKDCT-2610



HSRP

ActiveHSRP

Standby

HSRP Filtering

g gFHRP Filtering Solution Filter FHRP with combination of VACL and MAC route filter

Result: Still have one HSRP group with one VIP, but now ha

router at each site for optimal first-hop routing

HSRP

Active

HSRP Hellos HSRP

Routing Based Ingress Optimisatio

7/26/2019 BRKDCT-2610



VM= 10.10.10.1

Default GW = 10.10.10.100

ISP AISP B

Access

Agg

Data Centre A

LAN Extension

Prefix(EID)

Route Locator(RLOC)

10.10.10.1 A, B

10.10.10.2 A, B

… …

10.10.10.5 C, D

10.10.10.6 C, D

Ingress Tunnel

Rou ter (ITR)

Moved to C, D

Decap

3

IP_DA = 10.10.10.1

1

ETR

g g pLISP

A B C

IP_DA = B IP_DA = 10.10.10.1

IP_DA = 10.1

4

5Decap

7

IP_DA = 10.10.10.1

6Encap

2

VM=

Default G

IP_DA = 10.10.10.1

VM IP Address

10.10.10.1

7/26/2019 BRKDCT-2610



Access Layer

What Is FEX

7/26/2019 BRKDCT-2610



What Is FEX

FEX is an extension of theswitch that it connects to.

Nexus 5000 and Nexus 7000can be extended with a Nexus2000

FEX can be connected with

1/3/5/7/10m CX1, SR, LR, FET

FEX inherits the features of thedevice it is connected to

Nexus 2000 Designs

7/26/2019 BRKDCT-2610



Nexus 5000 Topologies (Nexus 2248TP & 2232PP)

Redundancy model – Dual Switch with redundantfabric

Provides isolation for Storage topologies (SAN ‘ A’

and ‘B’)

Port Channel and Pinning supported for FabricLink

vPC Supportedwith up to 2 x 8

links

LocalEtherchannelwith up to 8

links

FCoE Adapterssupported on 10G

N2K interfaces

Straight Through

Redundancy model – Single s‘supervisor ’ for fabric, data coplanes

No SAN ‘ A’ and ‘B’ isolation (Vsufficient in the future?)

Dual Homed

Nexus 2000 DesignN 7000 T l i (N 2248TP & 2232PP)

7/26/2019 BRKDCT-2610



Nexus 7000 Topologies (Nexus 2248TP & 2232PP)

LocalEtherchannelwith up to 8

links

NIC Teaming:TLB/ALB

Nexus 2248TP & 223

Fabric links supported on N7K-M132XP-12& N7K-M132XP-12L

Local port channel support(Future release)

No support for DCB and F

switch fabric ports not DCB

Nexus 2000 DesignT l i N t St

7/26/2019 BRKDCT-2610



Topologies – Next Steps

Redundancy model – Dual Switch (eachswitch supports redundant supervisors)

Future release

Nexus 5000Future

Redundancy model – Singdual ‘supervisor ’, fabric, lincontrol & management pla

MCECEtherchannelwith up to 16

links

Nexus 7000 – vPC

Current Data Centre ArchitectureWh I th Ed ?

7/26/2019 BRKDCT-2610



NIC

OperSyste

Device

Where Is the Edge?

The Data Centre Edge has historicallybeen well defined from a technical

and operational perspective

There have always been exceptionsto this rule but they were usuallyspecial cases and often involveddedicated access layer designs

The location of the edge is moving

–Hypervisor Virtual Switches

–SR-IOV

–FCoE

Edge of the Netwo

Eth

2/12

The Evolving Data Centre ArchitecturT h l Di t Vi t li ti

7/26/2019 BRKDCT-2610



Technology Disruptor - Virtualisation

0

2,500,000

5,000,000

7,500,000

10,000,000

12,500,00015,000,000

17,500,000

20,000,000

2005 2006 2007 2008 2009 2010 2011 2012 201Virtualized Non-Virtualized

Source: ID

Tipping Point

Traditional

1 Application…

...1 Server

Transition

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

Virtualised

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

App

OS

Current Data Centre ArchitectureH i S it h Wh I th Ed ?

7/26/2019 BRKDCT-2610



pNIC

Hypervisor virtualisation

resour

Hypervisor vSwitch—Where Is the Edge?

Hypervisor based compute virtualisationmoves the edge ofthe Fabric

PCI-E bus and storage and networkconnectivity resources are virtualised

–vSwitch

–VMFS (VMWare)

–NPV (provides FC SAN virtualisation)

With a shift in the edge of the fabriccomes a change in the operationalpractices and fabric design requirements

VNIC

VETH

Eth2/12

Unified FabricIEEE DCB

7/26/2019 BRKDCT-2610



IEEE DCB

Standard / Feature Status of the StandardIEEE 802.1QbbPriority-based Flow Control (PFC)

Done! And we are compliant!

IEEE 802.3bdFrame Format for PFC

Done! And we are compliant!

IEEE 802.1QazEnhanced Transmission Selection (ETS) andData Centre Bridging eXchange (DCBX)

Just completed WG; mid-March 2011

IEEE 802.1Qau Congestion Notification Done!

IEEE 802.1Qbh Port Extender In first working group ballot (which is the nesuccessful task group ballot as indicated in slide). Expect to complete in 6-12 months.

Developed by IEEE 802.1 Data Centre Bridging Task Group

CEE (Converged Enhanced Ethernet) is an informal group o

companies that submitted initial inputs to the DCB WGs.

Priority Flow ControlFibre Channel over Ethernet Flow Control

7/26/2019 BRKDCT-2610



Fibre Channel over Ethernet Flow Control

P a c k e t

R _RDY

Fibre Channel

Transmit Queues Ethernet LinkReceiv

One

Two

Three T

Four

Five

Seven S

Eight E

Six

STOP PAUSE

B2B Credits

Enables lossless Ethernet using PAUSE based on a COS as defined in 80

When link is congested, CoS assigned to FCoE will be PAUSEd so traffic

dropped Other traffic assigned to other CoS will continue to transmit and rely on up

protocols for retransmission

Enhanced Transmission StandardBandwidth Management IEEE 802 1Qaz

7/26/2019 BRKDCT-2610



Offered Traffic

t1 t2 t3

10 GE Link Realised Tra

3G/s HPC Traffic

3G/s

Storage Traffic

3G/s

3G/s

LAN Traffic

4G/s

3G/s

t1 t2

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Bandwidth Management -- IEEE 802.1Qaz

Required when consolidating I/O – It’s a QoS problem

Prevents a single traffic class of “hogging” all the bandwidth and starvi

classes When a given load doesn’t fully utilise its allocated bandwidth, it is ava

other classes

Helps accommodate for classes of a “burtsy” nature

Data Centre Bridging eXchangeControl Protocol the “handshake”

7/26/2019 BRKDCT-2610



Negotiates Ethernet capability’s : PFC, ETS, CoS values bDCB capable peer devices

Simplifies Management : allows for configuration and distrof parameters from one node to another

Responsible for Logical Link Up/Down signalling of EthernFibre Channel

DCBX negotiation failures result in:

per-priority-pause not enabled on CoS values

vfc not coming up – when DCBX is being used in FCoenvironment

Control Protocol – the handshake

Fibre Channel over EthernetWhat enables it?

7/26/2019 BRKDCT-2610



What enables it?

10Gbps Ethernet

Lossless Ethernet –Matches the lossless behaviour guaranteed in FC by B2B c

Ethernet jumbo frames

–Max FC frame payload = 2112 bytes

E t h e r n e t

H e a d e r

F C o E

H e a d e r

F C

H e a d e r

FC Payload

Same as a physical FC fra

Control information: version, ordered sets

Normal ethernet frame, ethertype = FCoE

FCoE Building BlocksThe New Buzzword ”Unified”

7/26/2019 BRKDCT-2610



Unified I/O – using Ethernet as the transport medium in allenvironments -- no long needing separate cabling options

and SAN networks Shared Wire – a single DCB Ethernet link actively carr

LAN and Storage (FC/FCoE/NAS/iSCSI) traffic simult

Dedicate Wire -- a single DCB Ethernet link capable oall traffic types but actively dedicated to a single traff

traffic engineering purposes Unified Fabric – An Ethernet Network made up of “Unified

everywhere: all protocols – network and storage –transversimultaneously

The New Buzzword… Unified

Fibre Channel over Ethernet Port Typ

7/26/2019 BRKDCT-2610



VE_Port

VF_Port

VF_Port

VE_Port

VN_Port

VN_Port

Fibre Channel over Ethernet Switch

FC

N

SwVF_Port VNP_PortFCF

Switch

End

Nod

End

Nod

FCoE Switch : FCF

Unified Fabric DesignUnified Edge

7/26/2019 BRKDCT-2610



Unified Edge The first phase of the Unified Fabric evolution

design focused on the fabric edge

Unified the LAN Access and the SAN Edge by

using FCoE

Consolidated Adapters, Cabling and Switching atthe first hop in the fabrics

The Unified Edge supports multiple LAN andSAN topology options

Virtualized Data Centre LAN designs

Fibre Channel edge with direct attachedinitiators and targets

Fibre Channel edge-core and edge-core-edge designs

Fibre Channel NPV edge designs

The Unified E

Fabric ALAN Fabric

F

Nexus 5000

FCF – NPV Mode

Unified Fabric DesignUnified Edge

Nexu

both d

7/26/2019 BRKDCT-2610



PCIe

E T H

F C

Link

1 0 G b E

1 0 G b E

Unified Edge

Converged Network Adapter (CNA) presentstwo PCI address to the Operating System (OS)

OS loads two unique sets of drivers andmanages two unique application topologies

Server participates in both topologies since ithas two stacks and thus two views of the same‘unified wire’

SAN Multi-Pathing provides failoverbetween two fabrics (SAN ‘ A’ and SAN

‘B’)

NIC Teaming provides failover within thesame fabric (VLAN)

FC Driver

bound to FC

HBA PCI

address

Unified Wire

shared by both

FC and IP

topologies

Nexus 5000

FCF-A

Operating Sy

Fibre Channel

Drivers

Unified Fabric with FCoEFCoE Design

7/26/2019 BRKDCT-2610



FCF

VLA

VLAN 10,20

FCoE Design

A VLAN is dedicated for every VSAN in the fabric

The VLAN is signaled to the hosts over FIP

The FCoE controller in the host tags allsubsequent FIP login and FCoE frames with thesignaled FCoE VLAN

This does‘not ’require trunking to be enabled atthe host driver as tagging is performed by theCNA

All ports in the FCoE network have to be enabledfor trunking to be able to carry VLAN taggedframes

Isolated Edge switches for SAN ‘ A’ and ‘B’ andseparate LAN switches for NIC 1 and NIC 2(standard NIC teaming)

! VLAN 20 is dedicated for V(config)# vlan 20(config-vlan)# fcoevsan2

SAN

VSAN 2

Unified Fabric with FCoEFCoE Design

7/26/2019 BRKDCT-2610



MCEC results in diverging LAN and SAN highavailability topologies

–FC maintains separate SAN ‘ A’ and SAN‘B’

topologies –LAN utilises a single logical topology

In vPC enabled topologies in order to ensurecorrect forwarding behaviour for SAN trafficspecific design and forwarding rules must befollowed

While the port-channel is the same on N5K-1

and N5K-2, the FCoE VLANs are different

vPC configuration works with Gen-2 FIPenabled CNAs ONLY

FCoE VLANs are ‘ not ’ carried on the vPCpeer-link

FCoE and FIP ethertypes are ‘ not ’ forwardedover the vPC peer link

SAN

Direct Attach vPC T

N5K1

FCoE Design

MCEC f

VLAN 10,3

VLAN 10,20

vPCVLA

Virtual Expansion Ports (VE_Ports)

7/26/2019 BRKDCT-2610



FCoE © 2011 Cisco and/or its affiliates All rights reserved Cisco Confidential

•Creates a standards based FC

•No further standards or protoc

for implementing “multihop” F

•Scalable Solution

Supports up to 7 hops – same

10,000 logins per fabric – sam

FC

E

E

FCoE

VE

VE

E_Ports

with FC

VE_Ports

with FCoE

FCoE Multi-Tier Fabric DesignExtending FCoE past the Unified Edge

7/26/2019 BRKDCT-2610



Extending FCoE past the Unified Edge Extending FCoE Fibre Channel fabrics beyond

direct attach initiators can be achieved in twobasic ways

Extend the Unified Edge Add DCB enabled Ethernet switches

between the VN and VF ports (stretch the‘link’ between the VN_Port and the VF_Port)

Extend Unified Fabric capabilities into the SANCore

Leverage FCoE wires between FibreChannel switches (VE_Ports)

What design considerations do we have whenextending FCoE beyond the edge?

High Availability

Oversubscription for SAN and LAN

Ethernet layer 2 and STP design

Fabric LAN Fabric

Fibre Channel Aware DeviceFCoE NPV

7/26/2019 BRKDCT-2610



What does an FCoE-NPV device do?

”FCoE NPV bridge" improves over a "FIPsnooping bridge" by intelligently proxying FIPfunctions between a CNA and an FCF

Active Fibre Channel forwarding and securityelement

FCoE-NPV load balance logins from the CNAsevenly across the available FCF uplink ports

FCoE NPV will take VSAN into account whenmapping or ‘pinning’ logins from a CNA to an

FCF uplink Emulates existing Fibre Channel Topology (same

mgmt, security, HA)

Avoids Flooded Discovery and Configuration (FIP)

Fibre Channel Configuration

and Control Applied at the

Edge Port

Proxy FCoE VLAN

Discovery

Proxy FCoE FCF

Discovery

FCoE Multi-Tier Larger Fabric Multi-Hop Topologies

7/26/2019 BRKDCT-2610



Servers, FCoEattached Storage

g p p g

Multi-hop edge/core/edge topology

Core SAN switches supporting FCoE

N7K with DCB/FCoE line cards MDS with FCoE line cards (Sup2A)

Edge FC switches supporting either

N5K - E-NPV with FCoE uplinks tothe FCoE enabled core (VNP toVF)

N5K or N7K - FC Switch withFCoE ISL uplinks (VE to VE)

Scaling of the fabric (FLOGI, …) willmost likely drive the selection of whichmode to deploy

N7e

Serv

VE

Edge FCFSwitchMode

VE

Cisco Nexus 1000V Components

7/26/2019 BRKDCT-2610



A B FC D E

vCenter Serv

Virtual Ethernet Mo

Replaces Vmware’s vi

Enables advanced swit

on the hypervisor Provides each VM with

“switch ports”

Virtual Supervisor Module(VSM)

CLI interface into the Nexus 1000V

Leverages NX-OS

Controls multiple VEMs as a singlenetwork device

Port Profile: Network Admin View

7/26/2019 BRKDCT-2610



n1000v# show port-profile name WebProfile

port-profile WebServers-PP

description:

status: enabledcapability uplink: no

system vlans:

port-group: WebServers

config attributes:

switchport mode access

switchport access vlan 110

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 110

no shutdown

assigned interfaces:

Veth10

Support Comma

Port managem

VLAN

PVLAN

Port-channel

ACL

Netflow

Port Security

QoS

Port Profile: Server Admin View

7/26/2019 BRKDCT-2610



Connectivity Best Practices

7/26/2019 BRKDCT-2610



vSphere

VM VM VM VM

If the upstream switch can be clu

(VPC, VBS Stack, VSS) use LAC

vSphere

VM VM VM VM

If the upstream switch can NOT be

clustered use MAC-PINNING

What is vPath ?Nexus

vPath

7/26/2019 BRKDCT-2610



vPath is intelligence build into Virtual Ethernet Module (VE

N1KV (1.4 and above)

vPath has two main functions:

a. Intelligent Traffic Steering to VSG

b. Offload the processing from VSG to VEM vPath is Multitenant Aware

Leveraging vPath enhances the service performance by mthe processing to Hypervisor

e us

Virtual Security GatewayIntel l igent Traff ic Steering w ith vPath

7/26/2019 BRKDCT-2610



g g

Nexus 1000V

Distributed Virtual Switch

VM VM VM

VM VM

VM

VM VM VM

VM

VM

VM VM VM

VM VM VMVM

VM

vPath

Initial Packet

Flow1 Flow Access Control

(policy evaluation)

2

Decision

Caching3

4

Virtual Security GatewayPerformance Acceleration w ith vPath

7/26/2019 BRKDCT-2610



Performance Acceleration w ith vPath

Nexus 1000V

Distributed Virtual Switch

VM VM VM

VM VM

VM

VM VM VM

VM

VM

VM VM VM

VM VM VMVM

VM

vPath

Remaining

packets from flow

ACL offloaded to

Nexus 1000V

(policy enforcement)

VSG: What Problem is Being Solved ?

7/26/2019 BRKDCT-2610



App

OS

A

O

App

OS

VM-to-VM traffic

Control inter-VM traffic

Address new blind spot

Mobility Transparent Enforcement

VLAN-agnostic OperationPolicy based

Administrative SegregationServer • Network • Security

Non-Disruptive AdministrationMitigate Operational errors between teams

7/26/2019 BRKDCT-2610



Network Admin SecuServer Admin

vCenter Nexus 1KV V

Mitigate Operational errors between teams

Security team defines security policies

Networking team binds port-profile to VSG service profile

Server team Assigns VMs to Nexus 1000V port-profiles

VSG Deployment Scenario – N1KV

7/26/2019 BRKDCT-2610



Standby

VSG

Active

VSG

…

Hypervisor N1KV VEMvPath



…

VSG is deployed to protect multiple hosts

Nexus 1000v is deployed with VEM having vPath intelli

Securing Virtual Desktops (Use Case)

7/26/2019 BRKDCT-2610



Persistent virtualworkspace for the doctor

Flexible workspace forDoctor ’s assistant

Maintain compliance

while supporting ITconsumerisation

RecordsHealthcare Portal Databas

Server Zones

AssistantIT Admin Docto

HVD Zones

Doctor

iT Admin Network

Virtual Security Gateway (VS

Cisco AnyConne

ASA

Summary Discussed Current Data Centre Challenges

7/26/2019 BRKDCT-2610



g

Reviewed solutions to accomplish active / active Laye

forwarding paths Reviewed solutions for active / active FHRP

Workload mobility at scale within a Data Centre as weacross Data Centre’s

Access layer solutions for 100Mb, 1GbE, 10GbE, Uniand Storage Integration with a standards based appro

Virtual access layer networking and security benefits achieve the dynamic elements of server virtualisation

7/26/2019 BRKDCT-2610



Q & A

Complete Your Online Session Evalua

7/26/2019 BRKDCT-2610



Complete your session evaluation:

Directly from your mobile device by visitingwww.ciscoliveaustralia.com/mobile and loginby entering your badge ID (located on thefront of your badge)

Visit one of the Cisco Live internet stationslocated throughout the venue

Open a browser on your own computer toaccess the Cisco Live onsite portal

http://www.ciscoliveaustralia.com/mobile

http://www.ciscoliveaustralia.com/mobile

7/26/2019 BRKDCT-2610


Documents

BRKDCT-2610