© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1

Nexus 1000V Deployment Scenarios

Dan HerseySteve Tegeler

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 2

Cisco Nexus 1000V Components

VMW ESX

Server 3VM #9

VM #12

VM #11

VM #10

VEMVMW ESX

Server 2VM #5

VM #8

VM #7

VM #6

VEMVMW ESX

Server 1VM #1

VM #4

VM #3

VM #2

VEM

Virtual Ethernet Module(VEM)Replaces existing vSwitchEnables advanced switching capability on the hypervisorProvides each VM with dedicated “switch ports”

Virtual Supervisor Module(VSM)CLI interface into the Nexus 1000VLeverages NX-OS 4.01Controls multiple VEMs as a single network device

Virtual Center

Nexus 1000V

VSM

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 3

Cisco Nexus 1000VFaster VM Deployment

VMW ESX

Server

VMW ESX

Server

Cisco Nexus 1000V

VM #1

VM #4

VM #3

VM #2

VM #5

VM #8

VM #7

VM #6

VM Connection PolicyDefined in the networkApplied in Virtual CenterLinked to VM UUID

Defined PoliciesWEB Apps

HR

DB

Compliance

Cisco VN-Link—Virtual Network LinkPolicy-Based

VM ConnectivityNon-Disruptive

Operational ModelMobility of Network

& Security Properties

Virtual Center

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 4

Cisco Nexus 1000VRicher Network Services

VMW ESX

Server

VMW ESX

Server

Cisco Nexus 1000V

VM #5

VM #8

VM #7

VM #6

VM #4

VM #3

VM #2

VM #1

VM #4

VM #3

VM #2

VM #1

VN-Link Property MobilityVMotion for the networkEnsures VM securityMaintains connection stateVirtual

Center

VMs Need to MoveVMotionDRSSW Upgrade/PatchHardware Failure

Policy-Based VM Connectivity

Non-DisruptiveOperational Model

Mobility of Network & Security Properties

VN-Link: Virtualizing the Network Domain

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 5

Cisco Nexus 1000VIncrease Operational Efficiency

VMW ESX

Server

VMW ESX

Server

Cisco Nexus 1000V

VM #5

VM #8

VM #7

VM #6

VM #4

VM #3

VM #2

VM #1

Network BenefitsUnifies network mgmt and opsImproves operational securityEnhances VM network featuresEnsures policy persistenceEnables VM-level visibility

Policy-Based VM Connectivity

Non-DisruptiveOperational Model

Mobility of Network & Security Properties

VN-Link: Virtualizing the Network Domain

Virtual Center

Server BenefitsMaintains existing VM mgmtReduces deployment timeImproves scalabilityReduces operational workloadEnables VM-level visibility

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 6

Nexus 1000V ‘Virtual Chassis’ Model

One Virtual Supervisor Module managing multiple Virtual Ethernet Modules

•Dual Supervisors to support HA environments

A single Nexus 1000V can span multiple ESX Clusters

SVS-CP# show moduleMod Ports Module-Type Model Status--- ----- -------------------------------- ------------------ ------------1 1 Supervisor Module Cisco Nexus 1000V active *2 1 Supervisor Module Cisco Nexus 1000V standby3 48 Virtual Ethernet Module ok4 48 Virtual Ethernet Module ok

--More--

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 7

Single Chassis Management

Upstream-4948-1#show cdp neighborCapability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID Local Intrfce Holdtme Capability Platform Port ID

N1KV-Rack10 Gig 1/5 136 S Nexus 1000V Eth2/2N1KV-Rack10 Gig 1/10 136 S Nexus 1000V Eth3/5N1KV-Rack10 Gig 1/12 136 S Nexus 1000V Eth21/2

A single switch from control plane and management plane perspective

Protocols such as CDP operates as a single switchXML API and SNMP management appears as a single ‘virtual chassis’

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 8

Virtual Supervisor Options

VSM VSM VSM

VSMVSMVSM

VSM Virtual ApplianceESX Virtual ApplianceSpecial dependence on CPVA serverSupports up to 64 VEMs

VMW ESX

Server 3VM #9

VM #12

VM #11

VM #10

VEMVMW ESX

Server 2VM #5

VM #8

VM #7

VM #6

VEMVMW ESX

Server 1VM #1

VM #4

VM #3

VM #2

VEM

VSM Physical ApplianceCisco branded x86 serverRuns multiple instances of the VSM virtual applianceEach VSM managed independently

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 9

Virtual Supervisor to Virtual Center

One way API between the VSM and Virtual CenterCertificate (Cisco self signed or customer supplied) ensures secure communicationsConnection is setup on the Supervisor

N1K-CP# show svs connections

Connection VC:IP address: 10.95.112.10Protocol: vmware-vim httpsvmware dvs datacenter-name: PHXLabConfigStatus: EnabledOperStatus: Connected

Nexus 1000V

VSMVirtual Center

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 10

Supervisor to Ethernet Module

Two distinct virtual interfaces are used to communicate between the VSM and VEM

•Control • Carries low level messages to ensure proper configuration of the VEM. • Maintains a 2 sec heartbeat what the VSM to the VEM (timeout 6 seconds)

•Packet •Carries any network packets between the VEM and the VSM such as CDP/LLDP

Must be on two separate VLANsSupports both L2 and L3 designs

VMW ESX

VM #1

VM #4

VM #3

VM #2

VEM

Nexus 1000V

VSM

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 11

Nexus 1000V Deployment Scenarios

Virtual Ethernet Modules

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 12

VEM Deployment Scenarios

VEM ConceptsLimits of VEM in Nexus 1000VInstallation of VEM

Port Types Defined & Addressing Mechanism for portsn1kv(Config t)# interface Module#/Eth#n1kv(Config t)# interface veth#

Spanning Tree Considerations/ConversationsGeneral Configuration Options for Traffic FlowSpecial Ports/VLANs used and I/O characteristics 1GE & 10GE deployment scenarios

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 13VSMVSMVSM

Virtual Ethernet Module BasicsVEM is a light weight (~10MB RAM) module that provides N1KV switching capability on the ESX host

Single VEM instance per ESX host

Relies on the VSM for configuration

Can run in last known good state without VSM connectivity

Some VMWare features will not work (Vmotion) when VSM is down

Must have VSM connectivity upon reboot to switch VM traffic

Virtual CenterVirtual Center

VMW ESXVMW ESXVMW ESX

Server 1Server 1

VMW ESXVMW ESXVMW ESX

Server 2Server 2

VMW ESXVMW ESXVMW ESX

Server 3Server 3

VMware vSwitch VMware vSwitch VMware vSwitch

VM #1VM VM #1#1

VM #4VM VM #4#4

VM #3VM VM #3#3

VM #2VM VM #2#2

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

VM #9VM VM #9#9

VM #12VM VM #12#12

VM #11VM VM #11#11

VM #10VM VM #10#10

VEMVEMVEM VEMVEMVEM VEMVEMVEM

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 14

Targeted Cisco Nexus 1000V Scalability

A single Nexus 1000V• 66 modules (2x Supervisors and 64x Ethernet Modules)

Virtual Ethernet Module: • 32 physical NICs

• 256 virtual NICs

Limit Per Nexus 1000V• 512 Port Profiles• 2048 physical ports • 8,192 virtual ports (vmknic, vswif, vnic)

Virtual Supervisor Virtual Supervisor -- StandbyStandby

VEMVEM

VEMVEM

VEMVEM

VEMVEM

VEMVEM

VEM VEM

VEMVEM

VEMVEM

VEMVEM

VEMVEM

Virtual Supervisor Virtual Supervisor -- ActiveActive

Nexus 1000V

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 15

VEM Distributed SwitchingUnique to each VEM

Data Plane MAC/Forwarding TableUpstream path

configuration (EtherChannel, pinning, etc)

Module # identification

Shared among all VEMs controlled by VSM

Control Plane (mgmt IP) Domain ID of N1K DVSPort Profile Configurationveth Interface Pool

Nexus 1000V

VSM

VMW ESX1

VEMModule 3

VMW ESX2

VEMModule 4

VMW ESX3

VEMModule n

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 16

Nexus 1000V

VSM

Installation of VEM

Current Virtual Ethernet Module code must be in lockstep with the ESX release version. Each time a new ESX server is deployed thecorrect VEM version must be loaded.

Automatic using VMWare Update Manager (VUM)

Or manual method with CLI command

VMW ESX VMW ESXVEM Module 3

VMW ESXVEM Module 4

Virtual CenterVirtual Center& VMWare Update Manager& VMWare Update Manager

I’m deploying a new ESX Server, do you have something for it?

VEM Module 5Yes I do!

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 17

Switching Interface Types - Eth

Physical Ethernet Ports (Network Admin Configuration) - NIC cards on each ESX server - Appears as a ‘Eth’ interface on a specific module in NX-OS

Example – ‘n1kv(Config t)# interface Eth3/1’-Module/Slot

- Module number is allocated when ESX is added to N1K- Server name to Module relationship can be found by issuing the ‘show module’ command

VM #2

VM #4

VM #1

VM #3

VMW esx1.cisco.com

VEMModule 3

VEMVEMModule 3Module 3

n1kv(Config t)# int eth3/1n1kv(Config t)# int eth3/2

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 18

Switching Interface Types - veth

Virtual Ethernet Ports- Virtual Machine/ESX facing ports - Appears as ‘veth’ within NX-OS- No “module” exists when configuring veth ports- Not being assigned to a specific module to simplifies VMotion

Example – ‘Veth68’

ESX1

VEMModule 5

VEMVEMModule 5Module 5

n1kv(Config t)# int veth1

VM #1

veth5

VM #2

veth6

VM #3

veth9

ESX2

VEMModule 6

VEMVEMModule 6Module 6

VM #4

veth68ServiceConsolevswif0 ve

th2

vmknic veth

3

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 19

Spanning Tree Considerations

There are none, but customers always want an explanation of whyBPDUs – if sent from an upstream switch, the Nexus 1000V drops themLoop prevention techniques will be used similar to the way VMWare provides todayIt will only learn MACs connected to a veth port on the local VEM by defaultIf destination is not on the local VEM, frame is forwarded out one of the physical interfacesThe best terminology to use with customers is to call the VEM a “Leaf Node”

VMW ESXVMW ESXVMW ESX

Server 2Server 2

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

Software Switch Software Switch Software Switch

A B

1 2 3 4

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 20

Configuration options for traffic flow

‘MAC Pinning’Embedded switch will determine and fix a path for each MAC address to use until a failure is detected

Virtual Port IDEssentially the same as MAC pinning, but based on the virtual NIC port @ FCS

VMW ESXVMW ESXVMW ESX

Server 3Server 3

VM #9VM VM #9#9

VM #12VM VM #12#12

VM #11VM VM #11#11

VM #10VM VM #10#10

VMW ESXVMW ESXVMW ESX

Server 2Server 2

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

Software Switch Software Switch Software Switch Software Switch Software Switch Software Switch

1 2

A B

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 21

Configuration options for traffic flow

HashingUsing some parameter to load balance across redundant links to an upstream switch or Cat6k VSS/Nexus vPC (i.e. MAC, IP, TCP, etc)

ManualManually configuring a path through a specific physical NIC to a specific vnic

VMW ESXVMW ESXVMW ESX

Server 3Server 3

VM #9VM VM #9#9

VM #12VM VM #12#12

VM #11VM VM #11#11

VM #10VM VM #10#10

VMW ESXVMW ESXVMW ESX

Server 2Server 2

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

Software Switch Software Switch Software Switch Software SwitchSoftware SwitchSoftware Switch

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 22

Channeling Techniques available with VMWare

NIC team load balancing algorithms based on either/or, not AND.

src MAC (MAC Pinning)virtual Port ID IP Hashing – equiv EtherChannelManual

VMWare doesn’t behave any differently if you are talking to the same upstream switch, or a different one. i.e. Hashing scenario

VMW ESXVMW ESXVMW ESX

Server 3Server 3

VM #9VM VM #9#9

VM #12VM VM #12#12

VM #11VM VM #11#11

VM #10VM VM #10#10

VMW ESXVMW ESXVMW ESX

Server 2Server 2

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

VMware vSwitch VMware vSwitch VMware vSwitch VMware vSwitch VMware vSwitch VMware vSwitch

A B

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 23

Channeling Techniques available with Nexus 1000V

Traffic flow is based on same principles as VMware, except N1KV can combine

src MAC (MAC Pinning)virtual Port ID EtherChannelManual

Primary Benefit of N1KV is the ability to pin traffic of specific VLANs to a certain upstream switch and provide EtherChannel

VMW ESXVMW ESXVMW ESX

Server 3Server 3

VMware vSwitch VMware vSwitch VMware vSwitch

VM #9VM VM #9#9

VM #12VM VM #12#12

VM #11VM VM #11#11

VM #10VM VM #10#10

VEMVEMVEM

VMW ESXVMW ESXVMW ESX

Server 2Server 2

VMware vSwitch VMware vSwitch VMware vSwitch

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

VEMVEMVEM

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 24

Possible Deployment Scenarios

Purpose of the following slides is to make you aware of the different architecture components of a ESX/N1KV environment

Any design sessions which leverage these slides before FCS, must come with the caveat that official best practices and recommendations may change

This is meant to start conversations and provideexamples of how it “could be”.

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 25

Priorities and I/O characteristics of Nexus 1000V VLANs & Virtual interfaces

Control VLAN – High Priority, Low BWUnique VLAN configured for VSM to VEM configuration, heartbeats, etc

Packet VLAN – Medium Priority, Low BWUnique VLAN configured for SUP level communication (IGMP, CDP, Netflow,

system logs – VACL/ACL, etc)

Vswif - Medium Priority, Low BWService Console/Management interface to the ESX Server – veth port

Vmknic – High or Low Priority & BWThe vmknic is used by the TCP/IP stack that services VMotion, NFS and software

iSCSI clients that run at the VMkernel level, and remote console traffic – veth port

Vnic – Priority & I/O characteristics depend on VMStandard VM data traffic – veth port

ESX1

VEMVEMVEM

VM #1

veth5

VM #2

veth6

VM #3

veth9

VM #4

veth68

Serv

ice

Con

sole

vsw

if0

veth

2

Vmkn

ic

veth

3Additional information & links found on this thread: http://communities.vmware.com/thread/136077?tstart=1775

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 26

1GE designPossible minimum

Multiple adapters for redundancy and throughput

1GE begs for traffic isolation as pipe can be filled

Minimum configs are four NICs (Two per EtherChannel) for Isolation and redundancy

ESX

VEMVEMVEM

VM #1

veth5

VM #2

veth6

VM #3

veth9

VM #4

veth68

Serv

ice

Con

sole

vsw

if0

veth

2

vmkn

ic

veth

3

Pinned TrafficN1KV ControlN1KV Packet

Service ConsolePossible VMkernel

Pinned TrafficVMs

Possible VMkernel

4Gb/s Total Bandwidth

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 27

1GE designA More Common Isolated Scenario

Multiple adapters for redundancy and throughput

Provide Isolation of different types of traffic

Guard against 1GE bottleneck

ESX

VEMVEMVEM

VM #1

veth5

VM #2

veth6

VM #3

veth9

VM #4

veth68

Serv

ice

Con

sole

vsw

if0

veth

2

vmkn

ic

veth

3

8Gb/s Total Bandwidth

N1KV ControlN1KV Packet

Service Console

VMs

VMkernel(IP Storage) VMkernel

(Vmotion)

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 28

Possible 10GE designsPin specific VLAN traffic to a specific uplink to enhance traffic isolation10GE likely to be enough BW for all trafficMinimum config would be two 10GE NICs for redundancy to two upstream switches

ESX

VEMVEMVEM

VM #1

veth5

VM #2

veth6

VM #3

veth9

VM #4

veth68

Serv

ice

Con

sole

vsw

if0

veth

2

vmkn

ic

veth

3

Pinned TrafficVMs

VMkernel

Pinned TrafficN1KV ControlN1KV Packet

Service ConsolePossible VMkernel

20 Gb/s Total Bandwidth

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 29

Your Feedback is important to us…We want to hear from you!

Please complete your Survey by going to the URL Please complete your Survey by going to the URL listed below:listed below:

http://iplatform.cisco.com/iplatform/

Event Name: Data Center SEVT

Session Name: Nexus 1000V Design Scenarios

© 2006 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 30