Operational Models for FCoE Deployments: Best Practices ... · PDF fileexperiences with Cisco customers who have successfully implemented FCoE. ... NOTE: Session focus is on Nexus

Operational Models for FCoEDeployments:

Best Practices and Examples

Jason Walker

Customer Solutions Architect

[email protected]

BRKSAN-2282

Session Objectives

• Review the operational challenges of adopting Fibre Channel over Ethernet (FCoE).

• Share Best Practices, Case Studies, and Configuration Examples based on experiences with Cisco customers who have successfully implemented FCoE.

• Help you determine the FCoE operational model for you organization to successfully share a Converged Network between LAN and SAN teams.

NOTE: Session focus is on Nexus 2K, 5K, 6K, 7K, MDS; not Nexus 9K or UCS.

• Recap of FCoE Technology

• Challenges for FCoE adoption

• Case Study: Maximum separation in a converged environment

• Nexus 7000/7700 Operational Model and Technology behind it

• Nexus 5000/6000 FCoE Deployment Models

• Case Study: Converged Access with Nexus 5000

• Nexus 5000 Operational Model and Technology behind it

• Unified Fabric Management

• FCoE Roadmap

Agenda









• FCoE Roadmap

Agenda

What Is FCoE? It’s Fibre Channel

Encapsulation of FC Frames over Ethernet

Enables FC to Run on a Lossless Ethernet Network

FCoE

Fibre

Channel

Traffic

Ethernet

Same Operational Model

Same Techniques ofTraffic Management

Same Managementand Security Models

Easy to Understand

Completely Based

on the FC Model

Same Host-to-Switch and Switch-to-

Switch Behavior as FC

Features Such as Order Delivery or

FSPF Load Balancing

WWNs, FC-IDs, Zoning, Nameserver,

RSCN

Standardized

in FC-BB-5

Converged Network AdaptersHost Bus Adapter (HBA) + Network Interface Controller (NIC)

First GenerationMid-2008

Multiple components

Full height/length Higher power

consumption

Fourth GenerationMid-2012

Next-gen Single chip

Less power consumption

Same support as HBAs

More than 1 million FCoE IOPS

Standard 10GEFCoE in Software

Brings the economy of scale of Ethernet to SAN

Native FCoE initiators in OS

Windows, Linux and vSphere 5

FREE

Open-

FCoE.org

Third Generation2010

Next-gen Single chip

Less power consumption

Same support as HBAs

Increased IOPS

mLoM CNA, OpenFCoEFCoE over 10GBaseT

Every Server is now 10GE enabled

FCoE is a software license or runs for free

10GBaseT FCoEsupport up to 30m

All Server Vendors (IBM, HP, Cisco, Dell) have certified CNAs and some have CNAoM.

Check the connectivity options for your CNA, they may vary - Twinax Active/Passive and 10GBaseT.

Second GenerationMid-2009

Single chip Half height/length Less than half the

power Same support as

HBAs

FCoE over 10G BASE-T

• Switch support for FCoE over 10GBASE-T:

• Nexus: 5596T, 6001T

• FEX: 2232TM-E, 2248TQ, 2232TQ

• CNA support for FCoE over 10GBASE-T:

• Cisco VIC 1225T for UCS C-Series: Cisco tested, interop tested with N6K-5K/N2K. OSM qualified.

• Intel X540T2 CNA: Cisco tested. Interop and OSM qualified.

• Emulex OneConnect OCe14102-UT: OSM qualified.

• Cables supported: Cat6a UTP/STP, Cat7 cables Qualified for up to 30 meters distance between 10GBASE-T host <-> switch.

FCoE Storage ArraysSupport from Major vendors, not a complete list

VendorMarketing

NameProduct Family

Max 10G FCoE

Ports

Max 8G FC

Ports

Max iSCSI

Ports

Unified

Storage

Symmetrix VMAX

20K/40K64 128 64x1G, 64x10G

VNX 7500/5x00 26 52 36x1G, 22x10G

Unified

Connect

FAS 6200/6000 40 64 44x1G, 40x10G

FAS 3200/3100 24 24 -

Virtual Storage VSP 96 192 -

Storage Center SC8000 10 16 10x1G, 10x10G

Unified Disk V7000 4 8 4x1G, 4x10G

For YourReference









• FCoE Roadmap

Agenda

LAN and SAN Team Intersection with FCoE

Dedicated HBAs

VLANs, QoS, Routing

GE, 10GE , LOM

Single Fabric High Availability

CLI, SNMP Based Tools

Modular, Top of Rack

Network Isolation and Security

VSANs, Zoning, I/O Performance

Dual Fabrics

Vendor Specific Management Tools

Director, Fabric Edge

Storage Virtualization

LAN Team

Host Adapter

Network Switch

Change Control

Troubleshooting

SAN Team

Top 5 Challenges in FCoE Deployments

Organization Structure

- More common discussion with medium and large companies

SAN and LAN Teams not under same management until Director or VP Level

Budget may comes from different departments

Separate teams that do not work together often in rare cases

Usually no push back from server team as they see HBA and Ethernet ports

Both teams should be part of testing

LAN Team is adopting Nexus, the foundation exist

FCoE License/Storage License much cheaper than dedicated hardware

1


Education for both LAN and SAN Teams

Education in general is a challenge

Provide cross-training building on existing knowledge

Great Example: Fiber Channel Networking for the IP Network Engineer and SAN Core Edge

Design Best Practices (BRKVIR-1121)

2

3 Insertion point

Retrofit an existing installation

Usually happens as part of a refresh, expansion or upgrade activity

New Servers or new Application will be added into a new or existing Data Center

Baseline current FC bandwidth consumption


Who maintains the converged network devices?

Software / Firmware Upgrades

With Nexus 5000/6000 using NPV Less dependency

Both Teams need to discuss together, change control process must be agreed:

- Example: LAN team request the upgrade based upon their requirements for more features

- SAN team would be given sufficient notice and time to:

- Review any open caveats and bugs

- Verify code version on Interoperability matrixes

- “Veto” the software / firmware version if there were issues that would impact the SAN

4


How does FCoE affect troubleshooting procedures?

LAN and SAN Team may utilize different tools

Separate support teams

Utilize tools that are aligned with Converged Networks – i.e. DCNM 5.2 and beyond

Establish processes that facilitate the communication between teams

Evolve the roles and responsibilities of the support teams, particularly Level 1 teams

5

Interoperability MatrixEcosystem readiness for FCoE, there are validated designs too

CISCO:

http://www.cisco.com/c/en/

us/td/docs/switches/datace

nter/mds9000/interoperabili

ty/matrix/intmatrx.pdf

EMC:

https://elabnavigator.emc.

com/vault/pdf/EMC_FCoE

_ESSM.pdf

IBM: http://www-03.ibm.com/systems/support/storage/ssic/interoperability.wss

HDS: http://www.hds.com/corporate/resources/

NetApp: http://now.netapp.com/NOW/knowledge/docs/olio/guides/tdps/tdps_interoperability_guide.xls

For YourReference









• FCoE Roadmap

Agenda

Ethernet

FC

Converged

FCoE Link

Dedicated

FCoE Link

Converged Access with Nexus 7k

• Shared Physical, Separate Logical LAN and SAN traffic at Access/Edge Layer

• Physical and Logical separation of LAN and SAN traffic at Aggregation/Core Layer

• Additional Physical and Logical separation of SAN fabrics

• Investment protection, Integrates into existing FC SAN

L2

L3

CNA

Fabric ‘A’ Fabric ‘B’

FCFCoE

Nexus 7000

Nexus 7700

MDS 9500

MDS 9700

MDS 9250i

Director-class SAN Edge

Fabric ‘A’

Fabric ‘B’

NX-OS 6.2(6) and above

supports FCoE with Phy-

port vPC on F2 and F2E.

Ethernet

FC

Converged

FCoE Link

Dedicated

FCoE Link

• Director-class SAN edge with ToRcabling

• NX-OS 7.2(1) supports FCoE on Nexus 7k using Nexus 2232PP FEX

• Benefits of both SAN edge models:

• Director class switching platform

• Efficient ToR cabling model (SAN fabric switch model)

L2

L3

CNA

Fabric ‘A’ Fabric ‘B’

FCFCoE

Nexus 7000

Nexus 7700

MDS 9500

MDS 9700

MDS 9250i

Director-Class SAN Edge with FEX

Fabric ‘A’

Fabric ‘B’

Converged Access with Nexus 7k

CNA

NEW!

Example

Director-Class SAN Edge with FEX

interface vfc4bind interface Ethernet101/1/4switchport trunk allowed vsan 3901no shutdown

FCoE-POC-N7706A-FCoE# sh int vfc4vfc4 is trunking

Bound interface is Ethernet101/1/4Hardware is EthernetPort WWN is 20:03:00:2a:6a:5c:53:bfAdmin port mode is F, trunk mode is onsnmp link state traps are enabledPort mode is TFPort vsan is 3901Speed is autoTrunk vsans (admin allowed and active) (3901)Trunk vsans (up) (3901)……

FCoE-POC-N7706A-FCoE# sh flogi database--------------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME--------------------------------------------------------------------------------vfc1 3901 0xb30060 20:00:00:0e:1e:36:b1:b8 10:00:00:0e:1e:36:b1:b8…vfc4 3901 0xb30020 10:00:00:90:fa:78:08:1f 20:00:00:90:fa:78:08:1fvfc5 3901 0xb30040 21:00:00:0e:1e:36:d2:33 20:00:00:0e:1e:36:d2:33

FCoE-POC-N7706A-FCoE# sh queuing interface e101/1/4Ethernet101/1/4 queuing information:Input buffer allocation:…Qos-group: 1frh: 3drop-type: no-dropcos: 3xon xoff buffer-size---------+---------+-----------21760 26880 43520

VLAN

3903

39

01

VLAN

3901

Maximum separation in a converged environmentLAN and SAN operated by two companies

Company A (Enterprise)

Retained server and storage operations

Concerned about Company B having access to data

Concerned about Company B operations impacting

Storage

Company B (Service Provider)

Concerned about Company A operations

impacting LAN

Have its own systems for Syslog, AAA, SNMP, NTP

Have its own systems for Syslog, AAA, SNMP,

NTP

Outsourced the LAN (IP) network to Company B

Storage

VDC

LAN

VDC

FCoE &

FIP

Ethernet

Creates a “virtual” MDS within the

Nexus 7000/7700

Only one such VDC can be created

Does not require Advanced License

(VDCs) but does count towards total

VDC count (4 with Sup1/2, 8 with

Sup2E)

Storage VDC

LAN

VDC

Storage

VDC

Converged I/O

FCoE

& FIPEthernet

Model for host interfaces, not ISLs

FCoE traffic is processed in the

context of the Storage VDC

Shared Interfaces

Exception to the rule allowing an

interface to exist in only one VDC

Splits traffic based on Ethertype

Ethernet VDC “owns” interface

Storage VDC sees the interface

as well

FCoE Initialization Protocol (FIP)

Ethertype 0x8914 and FCoE 0x8906

only are directed to the storage VDC.

All other Ethertypes are directed

toward the Ethernet VDC

Storage VDCImplementing FCoE on Nexus 7000/7700

VDC enabled companies to share the physical deviceIt allow us to solve Layer 8 issues…

Owned by Company BOwned by Company A

VDC 1

VDC 2

VDC 3

• No ports assigned (except

mgmt0)

• Initial device configuration

Used only for system maintenance

Default / Admin VDC

• FCoE ISLs to SAN

• Virtual FC Interfaces

SAN admin daily operations

SAN VDC

• FCoE host ports (shared)

• Non-FCoE host ports

• LAN uplinks to Agg Layer

LAN admin daily operations

LAN VDC

Role Based Access Control - RBACLimit access to operations on the device based on who you are

• Four default user roles

• network-admin - Complete read-and-write access to the entire device (only available in the Default VDC or Admin VDC)

• network-operator - Complete read access to the entire device (only available in the Default VDC or Admin VDC)

• VDC-admin - Read-and-write access limited to a VDC

• VDC-operator - Read access limited to a VDC

• You can create custom roles within a VDC

• VDCs on the same physical device do not share user roles

Users and Roles: Example of RBAC implementationVDC Role User Description

Default

or

Admin

network-admin

(exist by default

on Default VDC

or Admin VDC)

Example:

A-ADMIN

Super User.

Used for initial

configuration,

software upgrades

and VDC

management (i.e.

add more

interfaces).

Default

or

Admin

network-

operator

(exist by default

on Default VDC

or Admin VDC)

Example:

A-OPS

Read-only User,

read access to all

VDCs.

Not to be used.

Default

or

Admin

vdc-admin

(exist by default

on All VDCs but

it’s independent

from one VDC

to the other)

Example:

A-VDC-

ADMIN

Admin Users for

Default VDC.

They cannot take

actions that impact

the other VDCs.

Not able to switch

to other VDCs, not

able to reload the

switch, not able to

move interfaces

between VDCs.

VDC Role User Description

LAN vdc-admin

(exist by default

on All VDCs but

it’s independent

from one VDC

to the other)

Example:

B-LAN-

ADMIN

LAN administrator.

To be used by

Company B to

manage the LAN

VDC, similar to user

to manage a LAN

Switch.

Users cannot take

actions that impact

the other VDCs.

Not able to switch to

other VDCs, not able

to reload the switch,

but able to reload the

LAN VDC (like a LAN

Switch).

LAN vdc-operator

(exist by default

on All VDCs but

it’s independent

from one VDC

to the other)

Example:

B-LAN-

OPS

LAN Operator.

Read-only access

limited to the VDC,

can only read the

parameters

Used by B for Level 1

teams

LAN “Custom” Allow flexibility to

Company B for them

to create Roles that

match their existing

operational models

for LAN Switches.

VDC Role User Description

SAN vdc-admin

(exist by default

on All VDCs but

it’s independent

from one VDC

to the other)

Example:

A-SAN-

ADMIN

SAN Administrator.

To be used by

Company A to manage

the SAN VDC, similar

to user to manage a

SAN Switch.

Users cannot take

actions that impact the

other VDCs.

Not able to switch to

other VDCs, not able to

reload the switch, but

able to reload the SAN

VDC (like a SAN

Switch),

SAN vdc-operator

(exist by default

on All VDCs but

it’s independent

from one VDC

to the other)

Example:

A-SAN-

OPS

SAN Operator.

Read-only access

limited to the VDC, can

only read the

parameters, not able to

make any changes.

Used by A for Level 1

teams.

SAN “Custom” Allow flexibility to

Company A for them to

create Roles that

match their existing

operational models for

SAN Switches.

Initial ConfigurationWhat needs to be agreed before configuration?

• NX-OS Version to Run• http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html

• VDC Structure

• Default for Ethernet + Storage (2 VDCs) or

• Default for management + Non-Default Ethernet + Storage - 3 VDCs (Best Practice)

• Admin for management + Ethernet + Storage - 3 VDCs (Best Practice)

• The Host Facing Ports

• Amount of bandwidth on the converged links that is guaranteed for FCoE

• N7K default is 2G for FCoE and 8G for other traffic.

• The Uplink ports for LAN and SAN

• VLANs to be used for FCoE (from LAN VDC)

• Best Practice: Reserve a Range with same number of VSANs expected to be used

http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/recommended_releases/recommended_nx-os_releases.html

Initial ConfigurationThings to do in the DEFAULT OR ADMIN VDC

1) Install FCoE license

2) Install feature-set fcoe and associate the license with the F module

3) Enable feature lldp (not needed in the Admin VDC)

4) Enable system QoS and policy for FCoE

5) Create a VDC for LAN and SAN (Storage)7010-1(config)# VDC LAN

Note: Creating VDC, one moment please ...

7010-1(config)# VDC SAN type storage

7010-1(config-VDC)# limit-resource module-type fx

Use the “where” command

7010-1(config-VDC)# whereConf;VDC LAN admin@7010-1%default

Initial ConfigurationThings to do in the LAN VDC

6) Enable feature lldp

7) Enable feature lacp

LACP is best practice for port-channels

Don’t forget to enable LLDP on all VDCs!

Initial ConfigurationThings to do in the STORAGE VDC









4) Enable system QoS and default policy for FCoE

5) Create a VDC for LAN and SAN (Storage)

10) Allocate host interfaces to LAN VDC

- Host Facing Ports are allocated to LAN VDC

11) Allocate interfaces for uplink to SAN and LAN VDC

- Ports used for uplinks for FCoE ISL (SAN VDC)

- Ports used for uplinks towards LAN Aggregation Layer (LAN VDC)



Initial ConfigurationThings to do in the LAN VDC



12) Configure host ports within the LAN VDC7010-1-LAN(config)# interface Ethernet 7/19-22

7010-1-LAN(config-if-range)# switchport

7010-1-LAN(config-if-range)# switchport mode trunk

7010-1-LAN(config-if-range)# switch trunk allowed vlan 1,10-20

7010-1-LAN(config-if-range)# spanning-tree port type edge trunk







4) Enable system QoS and default policy for FCoE

5) Create a VDC for LAN and SAN (Storage)

10) Allocate interfaces to LAN VDC

- Host Facing Ports are allocated to LAN VDC

11) Allocate interfaces to SAN VDC

- Ports used for uplinks into the SAN fabrics (ISL)

13) Allocate FCoE VLANs and shared interfaces7010-1(config)# VDC SAN

7010-1(config-VDC)# allocate fcoe-vlan-range 40 from VDCs LAN

7010-1(config-VDC)# allocate shared interface ethernet 7/19-22

Ports that share the port group of the interfaces you have specified will be affected as well. Continue (y/n)? [yes]



Initial ConfigurationThings to do in the STORAGE VDC



14) Create a VSAN and a VLAN and Configure VSAN to VLAN Mapping

15) Configure FCoE port channel towards MDS

16) Configure VFC interface for VE port

17) Configure MDS for FCoE

18) Configure Host ports on SAN VDC and Create VFC interfaces and assign to VSAN7010-1-SAN(config)# interface eth7/19-22

7010-1-SAN(config-if-range)# switchport trunk allowed vlan 40

7010-1-SAN(config-if-range)# no shut

7010-1-SAN(config)# interface vfc 721

7010-1-SAN(config-if)# bind interface ethernet 7/21

7010-1-SAN(config-if)# switchport mode f

7010-1-SAN(config-if)# switchport trunk allowed vsan 40

7010-1-SAN(config-if)# no shut

7010-1-SAN(config-if)# vsan database

7010-1-SAN(config-vsan-db)# vsan 40 interface vfc721



Best Practice:

Formulate a method to map VFC

number to port. Example:

7/21 = vfc 721

FCoE ISLs on the MDSVirtual Expansion (VE) port considerations for SAN admin

interface eth7/31-32

switchport mode trunk

switchport trunk allowed vlan 40

channel-group group 350 force mode active

no shut

interface port-channel 350

no shut

interface vfc 350

bind interface port-channel 350

switchport mode e

switchport trunk allowed vsan 40

no shut

Nexus 7000 Storage VDC

interface ethernet 5/1-2

switchport mode trunk

switchport trunk allowed vlan 40

channel-group 350 force mode active

no shut

interface ethernet-port-channel 350

no shut

interface vfc 350

bind interface ethernet-port-channel 350

switchport mode e

switchport trunk allowed vsan 40

no shut

MDS 9500 / MDS 9700 / MDS 9250i

VE port = Expansion port in an FCoE network. You can bind a VE port to a physical Ethernet port or an

Ethernet Port Channel. Equivalent to establishing a FC Port Channel between two MDS switches

FCoE Requirements on Nexus 7000/7700 and MDSHardware and Software

• Nexus 7000

• Supervisor 1 or Supervisor 2 / 2E (FCoE on F2 / F2E / F3 requires SUP2/2E)

• F1-Series 10G Module, F2 / F2E 10G Modules, F3 40G Module

• F2E Module with 2232PP FEX

• FCoE license, one per F1 / F2 / F2E / F3 module

• Optional License that enables IVR - one per chassis - N7K-SAN1K9

• Optional DCNM-SAN Adv Edition to manage FCoE on Nexus 7000 - DCNM-SAN-N7K-K9

• Nexus 7700

• Supervisor 2E

• F2E 10G Module, F3 40G Module

• F2E Module with 2232PP FEX

• FCoE license, one per F2E / F3 module

• Optional License that enables IVR - one per chassis - N77-SAN1K9

• Optional DCNM-SAN Adv Edition to manage FCoE on Nexus 7700 - DCNM-SAN-N7K-K9

For YourReference

• MDS 9500

• 8-Port FCoE line card (DS-X9708-K9)

• MDS 9700

• MDS 9710 48-port 10-Gbps FCoE Line Card

• MDS 9250i

• 8 x 10-Gbps FCoE ports included in base configuration

• Minimum Software Required

• 5.2(1) for Nexus 7000 F1 and MDS 9500

• 6.1(1) for Nexus 7000 F2, 6.1(2) for Nexus 7000 F2-E

• 6.2(2) for Nexus 7700 F2-E

• 7.2(1) for Nexus 7000/7700 F3

• 7.2(1) for Nexus 7000/7700 w/ 2232 FEX

• 6.2(5) for MDS 9250i

• 6.2(7) for MDS 97100 48-port 10-Gbps FCoE Line Card

FCoE Requirements on Nexus 7000/7700 and MDSHardware and Software (continued)

For YourReference









• FCoE Roadmap

Agenda

Converged Access FCoE DesignsNexus 5000/6000 Connectivity Models

SAN A SAN BLAN Fabric

FC

FC

NativeFC

DedicatedFCoEOR

FC Switch Mode

Or

NPV Mode

FCoE

FCoE Direct Attach to Nexus 5K/6K- Simplest and Most Deployed Model

- Recommended to use 5K/6K in NPV Mode

FCFFCF5K/6K 5K/6K

EthernetFibre ChannelDedicated FCoE LinkConverged Link

ProtocolData Rate

Gbps MB/s

8G FC 6.8 850

10G FCoE 10.0 1250

Nearly 50% More Bandwidth!

Converged Access FCoE DesignsNexus 5000/6000 + Nexus 2000 Connectivity Models

Nexus 10GE FEX


FCoE with Single-Homed FEX

Nexus10GE FEX

NativeFC

DedicatedFCoE

OR

5K/6K5K/6K


FCoE with Dual-Homed FEX

Consider FCoE

Traffic Path

Nexus10GE FEX

NativeFC

DedicatedFCoE

OR

5K/6K 5K/6KFCF FCFFCF FCF


L2

Leverage FC SAN to Ethernet DCB switches in Aggregation layer with Dedicated links

Maintain the A – B SAN Topology with Storage VDC and Dedicated links

Nexus 5K or 6K as Consolidated Access layer

Dedicated FCoE Ports between:

Access and Aggregation

FC SAN and Aggregation

Zoning controlled by SAN A/B Core Switches

Multi-Hop FCoE DesignExtending FCoE through Aggregation

FCFCoE

AGG

Access

CORE

L3

5K/6K

MDS FC

SAN A

MDS FC

SAN B7K7K

5K/6K


MDS

9710

MDS

971048-port 10G

FCoE Module

Agg BW: 40G

FCoE: 20G

Ethernet: 20G

Different methods, Producing the same aggregate bandwidth

Dedicated Links provide additional isolation of Storage Traffic

Available on

Nexus 5K/6K and Nexus 7K

Available on Nexus 5K/6K

Not supported on Nexus 7K

Converged vs. Dedicated ISLsWhy support dedicated ISLs as oppose to Converged?

One wire for all traffic types

QoS guarantees minimum

bandwidth allocation

No Clear Port ownership

Desirable for DCI Connections

Converged

Dedicated wire for a traffic type

No Extra output feature

processing

Distinct Port ownership

Complete Storage Traffic

Separation

Dedicated

Reduced deployment times for each server by 25%

Saved 35% in Cap-ex and cut 90% of Power costs

“…utilizing the cores better, and increasing I/O with fewer cables and switches”

“…use our existing personnel to manage both IP and FCoE resources quickly and efficiently, increasing

their value within the company”

Adoption of Multi-hop FCoE For YourReference

Nexus

5000

Nexus

7000

CORE

SAN BSAN A

FCoE FCoE

EthernetFibre ChannelDedicated FCoEConverged Link

Multi-Hop FCoE End-to-End FCoE

http://www.cisco.com/c/dam/en/us/solutions/colla

teral/data-center-virtualization/boeing.pdf

Dynamic FCoEConvergence with Simplicity and Scale

Leaf

Spine


FabricPath multipathing

transparent to FCoE

FCoE-attached

HostFCoE-attached

Storage

FEX and Native FC

Attachment Also Supported

FabricPath

Nexus 6K

Nexus 5K/6K

Nexus 5600/6K

VE Link over FP

VF Link

Classical EthernetVF Link

Classical Ethernet

FCF FCF FCF

Dynamic FCoE Using FabricPath – NX-OS 7.0(1)N1(1)

FCF

Dynamic FCoEConvergence with Simplicity and Scale

Leaf

Spine


FabricPath multipathing

transparent to FCoE

FCoE-attached

HostFCoE-attached

Storage

FEX and Native FC

Attachment Also Supported

FabricPath

Nexus 5K/6K

VE Link over FP

VF Link

Classical EthernetVF Link

Classical Ethernet

FCF FCF FCF FCF

SAN BSAN ASAN BSAN A

http://blogs.cisco.com/datacenter/what-is-dynamic-fcoe/

Nexus 6KNexus 5600/6K









• FCoE Roadmap

Agenda

Case Study: Converged Access with N5KCommon Deployment Scenario

• Rack mount servers with TOR Nexus 5500/5600 switches

• Physical and Virtualized servers with 2-port CNAs

• Twinax 10GE cables for server 10G connectivity

• OM3 Fiber (for 8G FC and 10GE) to network/storage racks

• Dual-fabric MDS SAN Core

• SAN team primary management tool is Data Center Network Manager (DCNM)

FC

Case Study: Converged Access with N5KLAN and SAN Teams Working Together

• Nexus 5k running in FC NPV mode

• LAN team owns the physical Nexus 5k device

• SAN and LAN teams work together on initial configuration

• LAN Team :

• Uses custom lan-admin role for daily tasks on IP network

• Manages all Ethernet functions that don’t impact Storage

• Uses network-admin role for system upgrades and changes

Case Study: Converged Access with N5KLAN and SAN Teams Working Together

• SAN team:

• Uses custom san-admin role for all access

• Manages FC and VFC ports on Nexus 5k

• Performs FC zoning from SAN Core

• Coordinates with LAN team for system level changes

• For firmware upgrades, LAN team schedules, but SAN team:

• Is made aware of scheduled change

• Is given time to research firmware and test

• Can veto the upgrade if there are expected impacts to SAN

Domains of ManagementSharing an FCoE Switch

SAN Team Manages

Native FC ISLs to SAN

LAN Team Manages

Ethernet Uplinks to Data

Center Network

Network SAN

Shared Management:

LAN Team Manages physical

int, SAN Team manages vfc int

FCoE Hosts

VLAN OSPF

QoS LLDP

STP

vPC

BFD LACP NXOS

Zones/

Zonesets

VFC

Interfaces

FC /Device

Aliases

NPV +

TF Ports

SHARED NETWORK STORAGE

Role-Based Access Control (RBAC)Role Definitions for FCoE Deployments

• network-admin (predefined)

• Admin privileges for the converged switch

• lan-admin (user defined)

• IP network admin privileges (vlan, STP, ACLs, QoS, etc…)

• san-admin (predefined)

• SAN admin privileges (zoning, vsans, aliases, fspf, etc…)

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/

n5k_fcoe_ops_appA.html

FCoE and RBAC Configuration Template:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops_appA.html

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/operations/n5k_fcoe_ops_appA.html

Role cannot be changed and cannot be removed

A custom role can be created with san-admin as a baseline for more granular RBAC

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/521_n1_1/n5k_

ops_san_admin_role.html

SAN LAN

LAN

Admin

SAN

Admin

user: sanguy

FCDOMAIN

ZONING

FC ALIAS

FC INTERFACES

FSPF

FCOE-NPV

FABRIC BINDING

FCPING…

Pre-defined san-admin Role

NOTE: predefined san-admin role

allows configuration access to ALL

interfaces in the switch.

Added in

NX-OS 5.2

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/521_n1_1/n5k_ops_san_admin_role.html

Role-Based Access Control (RBAC)lan-admin Role (user defined)

role name lan-admin

description assuming fcoe vlan is 3010 and vfc9 has been

bound to eth1/9

rule 37 deny command config t ; spanning-tree vlan 3010

rule 36 permit command config t ; spanning-tree vlan *

rule 35 deny command config t ; spanning-tree *

…

rule 31 deny command config t ; vlan 3010 *

rule 30 deny command config t ; no vlan 3010 *

…

rule 1 permit read-write

interface policy deny

permit interface eth1/1-40

vlan policy deny

permit vlan 3010, 100-200

vsan policy deny

Basic Premise for lan-admin:

• Read access to everything

• Permit daily network admin tasks

• Prevent modifications to things

that can impact Storage traffic

Default VLAN, VSAN, and interface policy is

permit all. To restrict to a subset, change

policy to deny and add specific permits.

Role-Based Access Control (RBAC)Can the LAN admin change the SAN configuration?

N5k(config-if)# switchport trunk allowed vlan remove

3010

% VLAN permission denied Can’t Remove FCoE VLAN from Host Port

Can’t Configure Virtual FC PortN5k(config)# int vfc 9

% Interface permission denied

Can’t Modify VSAN DatabaseN5k(config)# vsan database

% Permission denied

No Zoning Commands AvailableN5k(config)# zone ?

^

% Invalid command at '^' marker.

No FC Commands AvailableN5k(config)# fc?

fcoe FCOE command

Can’t Shut Down FCoE Host PortN5k(config)# int eth 1/9

N5k(config-if)# shut

% Permission denied

Role-Based Access Control (RBAC)san-admin-c Role (user defined)

role name san-admin-c

description assuming fcoe vlan is 3010 and vfc9 has been

bound to eth1/9

rule 83 permit command config t ; vlan * ; fcoe *

rule 82 deny command config t ; vlan * ; *

…

rule 53 deny command config t ; interface * ; spanning-tree

bpduguard


cost *


guard *

…

rule 1 permit read-write

vlan policy deny

permit vlan 3010-3010

interface policy deny

permit interface fc2/1-8

permit interface Eth1/9

permit interface vfc 1-50

Basic Premise for san-admin-c:

• Read access to everything

• Permit daily SAN admin tasks

• Prevent modifications to all things

not related to Storage

Default VLAN, VSAN, and interface policy is

permit all. To restrict to a subset, change

policy to deny and add specific permits.

Role-Based Access Control (RBAC)Can the SAN admin change the LAN configuration?

N5k(config-if)# switchport trunk allowed vlan

remove 10

% Permission denied Can’t remove non-FCoE VLAN on Host Port

Can’t Configure non-FCoE Ethernet PortsN5k(config)# int eth 1/5

% Interface permission denied

Can’t Modify non-FCoE VLANsN5k(config)# vlan 10

% Permission denied

No Spanning Tree Commands AvailableN5k(config)# spanning?

^


No QoS Commands AvailableN5k(config)# class-map ?

^


System Management Services with FCoE

Nexus 5K/6K use shared system management

‒ AAA

‒ Syslog

‒ NTP

Shared System

Management

TACACS+ Syslog NTP

This is different than Nexus 7000 with Storage VDC, which

allows for distinct management services for LAN and SAN

Initial ConfigurationWhat needs to be agreed before configuration?

• NX-OS version to run

• http://www.cisco.com/c/en/us/support/switches/nexus-5000-series-switches/products-release-notes-list.html

• The Host Facing Ports

• Amount of bandwidth on the converged links that is guaranteed for FCoE

• N5K default is 5G for FCoE and 5G for other traffic.

• Uplink ports for LAN and SAN

• VLANs and VSANs to be used for FCoE

• Best Practice: Reserve the same range of VLAN/VSAN numbers

http://www.cisco.com/c/en/us/support/switches/nexus-5000-series-switches/products-release-notes-list.html

Initial ConfigurationWhat are the steps? What’s different?

1) Configure core SAN switch for NPIV

2) Install Storage Services (FC_FEATURES_PKG) license

3) Enable feature npv

4) Enable feature fcoe


N5k(config)# feature npvVerify that boot variables are set and the changes are saved. Changing to npv mode erases the current configuration and reboots the switch in npv mode. Do you want to continue? (y/n)

It’s recommended to enable

NPV first, as it will require a

reload of the switch.

Initial Configuration

6) Enable system QoS and policy for FCoE

7) Create VSAN and assign native FC interfaces to VSAN

8) Configure Fibre Channel interfaces to SAN

9) Create FCoE VLAN and map to VSAN

This step is only required on Nexus 55xx

platforms prior to release 5.1(3)N1(1).

The Nexus 50xx platforms have

FIP/FCoE in no-drop class by default.

N5k(config)# system qosN5k(config-sys-qos)# service-policy type qos input fcoe-default-in-policyN5k(config-sys-qos)# service-policy type queuing input fcoe-default-in-policyN5k(config-sys-qos)# service-policy type queuing output fcoe-default-out-policyN5k(config-sys-qos)# service-policy type network-qos fcoe-default-nq-policy

N5k(config)# vlan 3010N5k(config-vlan)# fcoe vsan 10

Best Practice:

Formulate a method to map

VSAN to VLAN. Examples:

VLAN 3010 = VSAN 10 or

VLAN 10 = VSAN 10

What are the steps? What’s different?


10) Configure host ports for FCoE traffic

11) Create virtual FC interface and bind to host interface

12) Assign vfc interface to VSAN

FCoE VLAN is defined here.

Native VLAN carries FIP traffic.

N5k(config)# interface ethernet 1/9-16N5k(config-if-range)# switchport mode trunkN5k(config-if-range)# switch trunk allowed vlan 3010N5k(config-if-range)# spanning-tree port type edge trunk

N5k(config)# interface vfc9N5k(config-if)# bind interface ethernet 1/9

N5k(config)# vsan databaseN5k(config-vsan-db)# vsan 10 interface vfc 9

Best Practice:

Formulate a method to assign vfc

number. Examples:

Eth 1/9 = vfc 9 or

Eth 1/9 = vfc 19



10) Configure host ports for FCoE traffic

11) Create virtual FC interface and bind to host interface

12) Assign vfc interface to VSAN

FCoE VLAN is defined here.

Native VLAN carries FIP traffic.

N5k(config)# interface ethernet 1/9-16N5k(config-if-range)# switchport mode trunkN5k(config-if-range)# switch trunk allowed vlan 3010N5k(config-if-range)# spanning-tree port type edge trunk

N5k(config)# interface vfc9N5k(config-if)# bind interface ethernet 1/9

N5k(config)# vsan databaseN5k(config-vsan-db)# vsan 10 interface vfc 9

Best Practice:

Formulate a method to assign vfc

number. Examples:

Eth 1/9 = vfc 9 or

Eth 1/9 = vfc 19


FCoE RequirementsHardware and Software Licensing

• Nexus 5548P / 5548UP / 5596UP

• FC Module (5548P only)

• Storage License (in 8-port increments)

• Optional FCoE NPV License (one per switch)

• Optional License for DCNM-SAN (one per switch)

• Nexus 5596T

• Same as above

• FCoE Support up to 30m distance over Cat 6a or 7 cabling

For YourReference

Unified Port

Native Fibre

Channel

FC Eth

Lossless

Ethernet

FCoE RequirementsHardware and Software Licensing

• Nexus 5672UP / 56128P / 6001

• Unified Port Module (56128 only)

• Storage License (in 16-port increments)



• Nexus 5624Q, 5648Q, 5696Q, 6004

• 20-port 10G Unified Port Module or 12-port 40G Module (5696Q and 6004)

• Storage License (in 16x10G or 4x40G increments)



For YourReference

Unified Port

Native Fibre

Channel

FC Eth

Lossless

Ethernet

FCoE RequirementsHardware and Software Licensing (continued)

• Nexus 2000 (FEX) FCoE Support

• 2232PP

• 2248PQ

• 2248UPQ

• 2232TQ (10GBaseT)

• 2248TQ (10GBaseT)

• 2232TM-E (10GBaseT)

• B22 HP (supported in HP Bladesystem c3000/c7000)

• B22 F (supported in Fujitsu PRIMERGY BX400/BX900)

• B22 DELL (supported in Dell PowerEdge M1000e)

• B22 IBM (supported in IBM Flex System Enterprise Chassis)

For YourReference

FCoE is not supported on the

2232TM Fabric Extender

Storage license only needed on parent

switch ports connecting to FEX

Which FCoE Access Model Should I Use?

SAN and LAN teams

require separate

managed devices?

Director class required

in the Edge Layer?

Use Nexus 7000/7700 in

the Edge/Access Layer

Use Nexus 5600/6000 in

the Edge/Access Layer

No No

YesYes

Cisco MDS in the

SAN Core?

Use native FC uplinks

from 5k/6k to SAN,

operating in NPV mode

Yes

Can FCoE module be

added to MDS?

Use FCoE uplinks from

5k/6k to MDS, operating

in FCoE NPV mode

Yes

No

No

Does customer desire

ToR Cabling Model?

Use Nexus 7000/7700

with FEX

Yes

Use Nexus 7000/7700

Direct Attach

No









• FCoE Roadmap

Agenda

Applications

ComputeSAN

Cisco Prime Data Center Network ManagerUnified LAN and SAN Operations for Nexus and MDS Platforms

LAN

• Converged monitoring

• Domain Dashboards

• Capacity Planning

• Wizard and template based provisioning

• Reduces complexity over non-integrated solutions

Unified Data Center FabricMulti-protocol Management

of NX-OS Platforms

• Unified discovery

• Dashboard views: summary, switches, hosts, storage enclosures

• VMpath Visibility

• Topology and path analytics

• Device groups and scoping

• Inventory and performance views: switches, modules, links, ports, …

• Configuration archive and restore

• SAN inventory, health and performance reports

Cisco Prime DCNM 7.1 Unified Web ClientLAN and SAN Inventory, Health, and Performance

Unified

Web Client

Windows

RHEL

Storage

Array

End-to-End Visibility

VMsESX

UCS or

other SERVER

MDSor NEXUS ISL MDS or NEXUS

STORAGE PORT

& LUN(s)

FC / FCoE

CNA Adapters

• Switch and Port Events

• Average and Peak TX/RX

• Switch CPU/Memory

• Traffic monitoring of ISLs and FCIP Links

• Discards and Errors

N5K-access-71 M97-75

N5K-access-70 M97-74

Cisco Prime DCNM Web ClientEnd-to-End Path Visibility Includes FCoE

FC / FCoE

FC / FCoE

FC / FCoE FC / FCoE

Unified

Web Client

• Detailed inventory and configuration management

• Supports all Nexus switches and technologies (FEX, VDC, vPC, Port Channel, FabricPath, …)

• LAN topology views

• L2 and security

• Wizards and template-based configuration

• Configuration archive and image management

• On-demand performance monitoring and thresholds

• Events

LAN Java

ClientCisco Prime DCNM LAN Java ClientComprehensive LAN Inventory, Topology, and Configuration

• SAN Fabric Management (FC, FCoE, FICON, iSCSI, FCIP, …)

• VSAN and Zoning management

• Wizard-based configuration (VSAN, IVR, FCoE, PortChannel, iSCSI, FCIP, NPV)

• Topology views and path display

• Path Redundancy Analysis

• Configuration archive and image management

• Troubleshooting tools

• Events and performance

Cisco Prime DCNM SAN Java ClientComprehensive SAN Fabric Management

SAN Java

Client

Data Center Network Manager FCoE Demo

https://www.youtube.com/watch?v=ZH07JafK89M

7 Minute Demo Available On YouTube:









• FCoE Roadmap

Agenda

FCoE RoadmapCisco Roadmap Disclaimer – The products and features described on this slide remain in

varying stages of development and will be offered on a when-and-if-available basis. This

roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability

for delay in the delivery or failure to deliver any of the products or features set forth in this

slide.

F3 10G FCoE SupportNexus 7000

Native FC Support on 2348UPQNexus 5000

2348UPQ FEX FCoE SupportNexus 7000

Nexus 5672UP16GNexus 5000

New UP Modules for 5624Q/5648QNexus 5000

Next 6 to 9 Months:

FCoE NPV (NX-OS Mode)Nexus 9000

Key Takeaways

Convergence with FCoE Delivers an Optimized Infrastructure

FCoE Deployment is About People & Process, not Technology

NX-OS RBAC Provides Separation of Duties

Storage VDC Provides an Additional Level of Isolation

Cisco continues to invest in FCoE

Related Sessions

• BRKSAN-2101 - FCoE for Small and Mid Size Enterprises

Monday 8:00am – 9:30am

• BRKVIR-1121 - Fiber Channel Networking for the IP Network Engineer and SAN Core Edge Design Best Practices

Tuesday 3:30pm – 5:00pm

• BRKSAN-2883 - Advanced Storage Area Network Design

Thursday 1:00pm - 2:30pm

• TECNMS-2008 - Data Center LAN and SAN Fabric Deployment, Troubleshooting & Management Systems Integration with Cisco Prime DCNM

Sunday 8:00am – 12:00pm

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

Thank you

Documents

Operational Models for FCoE Deployments: Best Practices ... · PDF fileexperiences with Cisco customers who have successfully implemented FCoE. ... NOTE: Session focus is on Nexus