Table  of  Contents  

Introduction  ..............................................................................................................................  2  

Nexus  1000V  Communication  Recommendation  Change  ..........................................  2  Hardware  and  Software  Environment  ......................................................................................  3  Cheat  Sheet  Tasks  .............................................................................................................................  4  

Nexus  5500  Configuration  ....................................................................................................  4  Nexus  5500  FCoE  Setup  ..................................................................................................................  4  Enabling  Storage  Services  ...........................................................................................................................  5  Creating  VSAN  and  FCoE  VLAN  .................................................................................................................  5  Configuring  FCoE  Storage  Ports  ................................................................................................................  6  

Nexus  5500  FEX  Configuration  .....................................................................................................  7  Nexus  5500  vPC  Configuration  .....................................................................................................  7  Creating  VLANs  on  Nexus  5548UP-­‐1  ......................................................................................................  8  Configuring  Nexus  5548UP-­‐1  for  vPC  feature  ....................................................................................  8  Configuring  Nexus  5548UP-­‐2  for  vPC  feature  ....................................................................................  9  Configuring  Nexus  5548UP-­‐1  for  vPC  for  Server1  ............................................................................  9  Configuring  Nexus  5548UP-­‐2  for  vPC  for  Server1  .........................................................................  10  Configuring  FCoE  Nexus  5548UP-­‐1  for  Server1  .............................................................................  10  Configuring  FCoE  Nexus  5548UP-­‐2  for  Server1  .............................................................................  11  

Nexus  1000V  Installation  and  Configuration  ..............................................................  12  Installing  Primary  and  Secondary  VSM  ...................................................................................  12  Configuring  Port-­‐Profiles  .............................................................................................................  19  Type  Ethernet  Port-­‐Profiles  ....................................................................................................................  20  Type  vEthernet  Port-­‐Profiles  ..................................................................................................................  21  

Adding  a  Server  as  a  VEM  .............................................................................................................  23  Adding  a  VEM,  Migrating  VMkernels  and  VSM  behind  the  VEM  .............................................  23  

Summary  ..................................................................................................................................  28  

 

Nexus  1000V  Configuration  Cheat  Sheet  –  VSM  in  Layer  3  mode    

Date: March 2012

Introduction This document will walk you through how to install and configure a Nexus 1000V with the virtual supervisor module (VSM) in Layer 3 (L3) mode. The following is the topology that will be used for the cheat sheet. Figure 1: Physical Topology

Since we will be utilizing Layer 3 mode for the VSM, there is no need for “Control” and “Packet” VLANs to be created and those interface VLANs will default to VLAN 1. Details on how that is configured will also be shown later in the documentation.

Nexus 1000V Communication Recommendation Change The Cisco Nexus 1000V has improved many of its capability and usability aspects since its first release. Two of the things that fall into the usability section are how to install the Nexus 1000V VSM and adding the VEM. Within the usability enhancement, the Nexus 1000V recommendation has changed from “Layer 2” to “Layer 3” mode for the communication between the VSM and VEM. This change in recommendation has come since the release of 4.2(1)SV1(4) and higher.

There are multiple reasons behind this change, the most critical being the ease of troubleshooting. When the VSM and VEM are connected in the same physical switch, the communication is done through the “Control” VLAN and troubleshooting is relatively simple. In most deployments, especially in blade server environments, there are multiple hops between the VSM and VEM. In layer 2 mode, troubleshooting tasks are done by both the network and the server team to resolve the communication (Control VLAN) problem. The server team will need to execute network commands, which has to be done on the ESX/ESXi server through command line interface. This will have to track mac-address, VLANs and other networking constructs to troubleshoot why the VSM is not able to communicate with the VEM. With Layer 3 mode for this communication, the VEM (vmkernel interface) needs to be able to ping the management interface of the VSM. Prior to installing the Nexus 1000V, the ESX/ESXi server is already utilizing layer 3 to communicate from its management interface to the vCenter Server. So using this same capability, the Nexus 1000V requires to have a vmkernel (recommended to be the management interface) interface to use for this layer 3 VSM to VEM communication. Figure 2: VSM to VEM Communication

Hardware  and  Software  Environment  The following are the hardware and software used for the cheat sheet Nexus Switches

• Nexus 5548UP and Nexus 2232PP o NX-OS version 5.1(3)N1(1)

• Nexus 1000V o NX-OS version 4.2(1)SV1(5.1)

Server Hardware/Software

• UCS C200 M2 o Emulex OCe11102 CNA be2net driver 4.0.355.1 o ESXi 5.0 build 469512 o vCenter 5.0 Server build 455964 o VMware Update Manager 5.0.0.8039

Storage Array

• EMC VNX 5300 o Firmware 5.31.000.5.502

Cheat  Sheet  Tasks  The following are the high level tasks that will be completed:

1. Nexus 5500 Configuration a. FCoE Setup b. FEX Setup c. vPC Setup

2. Nexus 1000V Installation and Configuration a. Installation of VSM as a VM (both primary and secondary) b. Configuring Port-Profile of type Ethernet (uplinks) c. Configuring Port-Profiles of type vEthernet d. Adding VEM

Nexus 5500 Configuration In this section, the Nexus 5500 switches will be configured to prepare for the necessary features needed for this solution. The key features that will be configured are:

• FCoE • FEX • vPC

Nexus  5500  FCoE  Setup  The figure below shows the details of the Nexus 5500 environment for FCoE: Figure 3: Nexus 5500 FCoE Topology

Note: The Nexus 5548UP-1 switch will be used for Fabric-A and Nexus 5548UP-2 will use Fabric-B. The first task is to enable “Storage Services” to allow the Nexus 5500 to provide Fibre Channel (FC) services as well as FCoE. Please follow the steps below to set up the Nexus 5500 for FC and FCoE services.

Enabling  Storage  Services  5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# feature fcoe FC license checked out successfully fc_plugin extracted successfully FC plugin loaded successfully FCoE manager enabled successfully FC enabled on all modules successfully Enabled FCoE QoS policies successfully

Note: With the release of NX-OS 5.1(3)N1(1), it is not necessary to manually configure the QoS settings for FCoE Class-of-Service (CoS) on the Nexus 5500s. It is automatically done for you unless there is a conflict with an existing QoS setting on the switch, in which case a manual configuration is needed.

Creating  VSAN  and  FCoE  VLAN  For our Fabric-A, we will create VSAN 10 for the environment and utilize VLAN 10 as our FCoE VLAN. Follow the steps below to complete these tasks.

5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# vsan database 5548up-1(config-vsan-db)# vsan 10 5548up-1(config-vsan-db)# exit 5548up-1(config)# vlan 10

5548up-1(config-vlan)# fcoe vsan 10

Configuring  FCoE  Storage  Ports  The following steps will walk you through how to configure the FCoE ports for the EMC VNX storage array.

Enabling  EMC  VNX  FCoE  Port.  5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# interface vfc123 5548up-1(config-if)# vsan database 5548up-1(config-vsan-db)# vsan 10 interface vfc123 5548up-1(config-vsan-db)# interface vfc123 5548up-1(config-if)# bind interface ethernet 1/23 5548up-1(config-if)# switchport trunk allowed vsan 10 5548up-1(config-if)# no shutdown 5548up-1(config-if)# interface ethernet 1/23 5548up-1(config-if)# switchport mode trunk 5548up-1(config-if)# switchport trunk allowed vlan 1, 10 5548up-1(config-if)# spanning-tree port type edge trunk Warning: Edge port type (portfast) should only be enabled on ports connected to a single host. Connecting hubs, concentrators, switches, bridges, etc... to this interface when edge port type (portfast) is enabled, can cause temporary bridging loops. Use with CAUTION 5548up-1(config-if)# show interface vfc123 vfc123 is trunking Bound interface is Ethernet1/23 Hardware is Ethernet Port WWN is 20:7a:00:05:73:ed:72:bf Admin port mode is F, trunk mode is on snmp link state traps are enabled Port mode is TF Port vsan is 10 Trunk vsans (admin allowed and active) (10) Trunk vsans (up) (10) Trunk vsans (isolated) () Trunk vsans (initializing) () 1 minute input rate 8 bits/sec, 1 bytes/sec, 0 frames/sec 1 minute output rate 56 bits/sec, 7 bytes/sec, 0 frames/sec 15 frames input, 1616 bytes 0 discards, 0 errors 16 frames output, 2000 bytes 0 discards, 0 errors last clearing of "show interface" counters never Interface last changed at Mon Dec 19 23:46:34 2011 5548up-1# show flogi database vsan 10 -------------------------------------------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------------------------------------------- vfc123 10 0xe50001 50:06:01:60:3e:a4:33:27 50:06:01:60:be:a0:33:27 Total number of flogi = 1.

Note: When configuring the FCoE port for the EMC VNX array, it is important to verify that the EMC array has allow the particular FCoE VLAN (ex: VLAN 10) to traverse that particular port. REPEAT THESE STEPS FOR NEXUS 5548UP-2 TO CONFIGURE FCoE Note: Make sure VSAN and VLAN are set for 11 and NOT 10

Nexus  5500  FEX  Configuration  This section will configure the Nexus 2232PP connected to the Nexus 5548UP. The following is a more detailed diagram for this section: Figure 4: Nexus 2232PP Configuration Details

5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# feature fex 5548up-1(config)# fex 101 5548up-1(config-fex)# interface port-channel 101 5548up-1(config-if)# switchport mode fex-fabric 5548up-1(config-if)# fex associate 101 5548up-1(config-if)# interface ethernet 1/21-22 5548up-1(config-if-range)# switchport mode fex-fabric 5548up-1(config-if-range)# fex associate 101 5548up-1(config-if-range)# channel-group 101 5548up-1(config-if-range)# show fex FEX FEX FEX FEX Number Description State Model Serial --------------------------------------------------------------------------------------------- 101 FEX0101 Online N2K-C2232PP-10GE SSI141902DK

REPEAT THESE STEPS FOR NEXUS 5548UP-2 TO CONFIGURE THE FEX Note: The FEX number for the Nexus 5548UP-2 should be 102 for the Nexus 2232PP.

Nexus  5500  vPC  Configuration  This portion of the configuration will focus on enabling and configuring virtual port-channels for the server uplinks to the Nexus 2232. The diagram below will provide the details that will be used for the vPC configuration.

Figure 5: Server vPC Topology Details

Creating  VLANs  on  Nexus  5548UP-­‐1  5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# vlan 51 5548up-1(config-vlan)# name Vmotion 5548up-1(config-vlan)# vlan 52 5548up-1(config-vlan)# name Web-Server 5548up-1(config-vlan)# vlan 53 5548up-1(config-vlan)# name Database-Server 5548up-1(config-vlan)# vlan 172 5548up-1(config-vlan)# name Management

REPEAT THE CREATION OF THE SAME VLANS ON THE NEXUS 5548UP-2

Configuring  Nexus  5548UP-­‐1  for  vPC  feature  5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# feature vpc 5548up-1(config)# feature lacp 5548up-1(config)# interface port-channel 1 5548up-1(config-if)# switchport mode trunk 5548up-1(config-if)# interface ethernet 1/15-16 5548up-1(config-if-range)# switchport mode trunk 5548up-1(config-if-range)# channel-group 1 mode active 5548up-1(config-if-range)# exit

5548up-1(config)# vpc domain 5 5548up-1(config-vpc-domain)# peer-keepalive destination 10.29.172.12 vrf management 5548up-1(config-vpc-domain)# interface port-channel 1 5548up-1(config-if)# vpc peer-link Please note that spanning tree port type is changed to "network" port type on vPC peer-link. This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance (which is enabled by default) is not disabled.

Configuring  Nexus  5548UP-­‐2  for  vPC  feature  5548up-2# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-2(config)# feature vpc 5548up-2(config)# feature lacp 5548up-2(config)# interface port-channel 1 5548up-2(config-if)# switchport mode trunk 5548up-2(config-if)# interface ethernet 1/15-16 5548up-2(config-if-range)# switchport mode trunk 5548up-2(config-if-range)# channel-group 1 mode active 5548up-2(config-if-range)# exit 5548up-2(config)# vpc domain 5 5548up-2(config-vpc-domain)# peer-keepalive destination 10.29.172.11 vrf management 5548up-2(config-vpc-domain)# interface port-channel 1 5548up-2(config-if)# vpc peer-link Please note that spanning tree port type is changed to "network" port type on vPC peer-link. This will enable spanning tree Bridge Assurance on vPC peer-link provided the STP Bridge Assurance (which is enabled by default) is not disabled.

 

Configuring  Nexus  5548UP-­‐1  for  vPC  for  Server1  5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# interface port-channel 1001 5548up-1(config-if)# switchport mode trunk 5548up-1(config-if)# spanning-tree port type edge trunk 5548up-1(config-if)# vpc 1001 5548up-1(config-if)# interface ethernet 101/1/1 5548up-1(config-if)# switchport mode trunk 5548up-1(config-if)# spanning-tree port type edge trunk 5548up-1(config-if)# channel-group 1001 mode active 5548up-1(config-if)# show port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SU) Eth LACP Eth1/15(P) Eth1/16(P) 101 Po101(SU) Eth NONE Eth1/21(P) Eth1/22(P) 1001 Po1001(SU) Eth LACP Eth101/1/1(P)

Configuring  Nexus  5548UP-­‐2  for  vPC  for  Server1  5548up-2# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-2(config)# interface port-channel 1001 5548up-2(config-if)# switchport mode trunk 5548up-2(config-if)# spanning-tree port type edge trunk 5548up-2(config-if)# vpc 1001 5548up-2(config-if)# interface ethernet 102/1/1 5548up-2(config-if)# switchport mode trunk 5548up-2(config-if)# spanning-tree port type edge trunk 5548up-2(config-if)# channel-group 1001 mode active 5548up-2(config-if)# show port-channel summary Flags: D - Down P - Up in port-channel (members) I - Individual H - Hot-standby (LACP only) s - Suspended r - Module-removed S - Switched R - Routed U - Up (port-channel) M - Not in use. Min-links not met -------------------------------------------------------------------------------- Group Port- Type Protocol Member Ports Channel -------------------------------------------------------------------------------- 1 Po1(SU) Eth LACP Eth1/15(P) Eth1/16(P) 102 Po101(SU) Eth NONE Eth1/21(P) Eth1/22(P) 1001 Po1001(SU) Eth LACP Eth102/1/1(P)

REPEAT THE CREATION OF THE vPC for Server2 Installation of ESXi 5.0 still needs to be completed. This paper will not show the installation process. For this, please refer to VMware’s documentation on installation of vSphere 5.0.

Configuring  FCoE  Nexus  5548UP-­‐1  for  Server1  5548up-1# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-1(config)# interface vfc 1001 5548up-1(config-if)# bind interface ethernet 101/1/1 5548up-1(config-if)# switchport trunk allowed vsan 10 5548up-1(config-if)# no shutdown 5548up-1(config-if)# vsan database 5548up-1(config-vsan-db)# vsan 10 interface vfc 1001 5548up-1(config-vsan-db)# show interface vfc 1001 vfc1001 is trunking Bound interface is Ethernet101/1/1 Hardware is Ethernet Port WWN is 23:e8:00:05:73:ed:72:bf Admin port mode is F, trunk mode is on snmp link state traps are enabled Port mode is TF Port vsan is 10 Trunk vsans (admin allowed and active) (10) Trunk vsans (up) (10) Trunk vsans (isolated) ()

Trunk vsans (initializing) () 1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 74 frames input, 10116 bytes 0 discards, 0 errors 80 frames output, 9208 bytes 0 discards, 0 errors last clearing of "show interface" counters never Interface last changed at Mon Jan 16 01:09:48 2012

REPEAT THE CREATION OF THE VFC for Server2 Below is an output of the fabric login database:

5548up-1# show flogi database vsan 10 -------------------------------------------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------------------------------------------- vfc123 10 0xe50001 50:06:01:60:3e:a4:33:27 50:06:01:60:be:a0:33:27 vfc1001 10 0xe50002 10:00:00:00:c9:a0:a1:93 20:00:00:00:c9:a0:a1:93 vfc1002 10 0xe50003 10:00:00:00:c9:a0:9f:1f 20:00:00:00:c9:a0:9f:1f Total number of flogi = 3.

 

Configuring  FCoE  Nexus  5548UP-­‐2  for  Server1  5548up-2# configure terminal Enter configuration commands, one per line. End with CNTL/Z. 5548up-2(config)# interface vfc 1001 5548up-2(config-if)# bind interface ethernet 102/1/1 5548up-2(config-if)# switchport trunk allowed vsan 11 5548up-2(config-if)# no shutdown 5548up-2(config-if)# vsan database 5548up-2(config-vsan-db)# vsan 11 interface vfc 1001 5548up-2(config-vsan-db)# show interface vfc 1001 vfc1001 is trunking Bound interface is Ethernet102/1/1 Hardware is Ethernet Port WWN is 23:e8:54:7f:ee:0c:08:bf Admin port mode is F, trunk mode is on snmp link state traps are enabled Port mode is TF Port vsan is 11 Trunk vsans (admin allowed and active) (11) Trunk vsans (up) (11) Trunk vsans (isolated) () Trunk vsans (initializing) () 1 minute input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 1 minute output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 55 frames input, 7704 bytes 0 discards, 0 errors 60 frames output, 6728 bytes 0 discards, 0 errors last clearing of "show interface" counters never

Interface last changed at Mon Jan 16 01:19:17 2012 REPEAT THE CREATION OF THE VFC for Server2

5548up-2# show flogi database vsan 11 -------------------------------------------------------------------------------------------------------------------- INTERFACE VSAN FCID PORT NAME NODE NAME -------------------------------------------------------------------------------------------------------------------- vfc123 11 0x610001 50:06:01:61:3e:a4:33:27 50:06:01:60:be:a0:33:27 vfc1001 11 0x610002 10:00:00:00:c9:a0:a1:91 20:00:00:00:c9:a0:a1:91 vfc1002 11 0x610003 10:00:00:00:c9:a0:9f:1d 20:00:00:00:c9:a0:9f:1d

Total number of flogi = 3.

In this setup, we will zone up the FCoE ports from the initiators to the FCoE targets. Even though there is a FC storage port, it is not necessary to add that to the zone. The following is the zoneset and zone created for this environment.

5548up-1# show zoneset active vsan 10 zoneset name J05-ZoneSet-A vsan 10 zone name C200M2-ESXi5-0 vsan 10 * fcid 0xe50001 [pwwn 50:06:01:60:3e:a4:33:27] * fcid 0xe50002 [pwwn 10:00:00:00:c9:a0:a1:93] * fcid 0xe50003 [pwwn 10:00:00:00:c9:a0:9f:1f] 5548up-2# show zoneset active vsan 11 zoneset name J05-ZoneSet-B vsan 11 zone name C200M2-ESXi-5-0 vsan 11 * fcid 0x610001 [pwwn 50:06:01:61:3e:a4:33:27] * fcid 0x610002 [pwwn 10:00:00:00:c9:a0:a1:91] * fcid 0x610003 [pwwn 10:00:00:00:c9:a0:9f:1d]

With the zoneset activated, the storage array is now able to see the UCS rack servers. Providing LUNs to the servers will not be shown here but a 500GB LUN is made available to both of the servers. The storage name for the ESXi servers is called “VNX-LUN0”.

Nexus 1000V Installation and Configuration This section will focus on installing and configuring the Nexus 1000V as a Virtual Machine (VM). Installation of both Primary and Secondary VSMs will be shown.

Installing  Primary  and  Secondary  VSM  With the release of Nexus 1000V 1.5, an Installer Application assists on installing the VSMs on the vSphere servers. The Installer Application is a part of the zip file for the release of 1.5, where the path is “Nexus1000v.4.2.1.SV1.5.1\VSM\Installer_App\Nexus1000V-install”. This cheat sheet will utilize this “Installer App” to deploy the Primary and Secondary VSMs. Please follow these steps to complete this task. Figure 6: Installer Application Wizard vCenter Credentials

1. Enter in the IP Address of the vCenter Server 2. Enter in the UserID to the vCenter Server 3. Enter in the Password for the “Administrator” 4. Click on “Next”

Figure 7: Selection of Host Where VSM Will Reside

Select the ESXi server and click on “Next”

Figure 8: VSM Creation Through OVA File

1. Click on the “Browse OVA” button and find the 1.5 OVA file, which should be “Nexus1000v.4.2.1.SV1.5.1\VSM\ Install\nexus-1000v.4.2.1.SV1.5.1.ova”

2. Leave the default redundancy to HA, which will install Primary and Secondary VSM

3. Provide a name for the virtual machine name for the VSMs and a “-1” and “-2” will be added to this name

4. Select the storage that the VSMs will reside on. Figure 9: VSM Networking Configuration

1. Select “L3: Configure port groups for L3 2. Select the radio button “Create New” for the Control port group 3. Create the port group “N1KV-Control, VLAN: 1” for the control interface 4. Select the radio button “Create New” for the Management port group. 5. Create the port group “N1KV-Mgmt, VLAN: 172” for the management

interface 6. Select the radio button for the L3 Connectivity to be “mgmt0” of the VSM 7. Type in the VLAN ID number that will be used in creating the L3

communication port-profile. This port-profile will be used for the vmkernel interface on the ESXi (VEM) that will do the L3 communication between the VSM and VEM, which is named “n1kv-L3”

8. Click on “Next”

Figure 16: VSM Configuration Information

1. Provide the VSM name 2. Enter in a secure password and re-enter the password for verification 3. Enter in the management IP Address, netmask and default gateway 4. Provide a unique domain id for the VSM instance 5. Select from the appropriate vCenter Datacenter Name from the drop down

box 6. Enter in vSwitch’s Native VLAN ID number, default should be VLAN 1 7. Click on “Next”

Figure 17: Review VSM Network Information

Figure 18: Installation of VSMs, Registering Extension and Creation of SVS Connection

Figure 19: Configure Migration

Select “No” to migrate the current server hosting the VSMs to become a VEM. Then click on “Finish”. Figure 20: Completion of Installer Application

Note: The installer application places both primary and secondary VSM on the same host. As a best practice, the primary and secondary VSM should reside on different host. Verify that both primary and secondary VSMs are installed and the svs connection is configured. Execute the following commands:

J05-L3-VSM# show module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 1 0 Virtual Supervisor Module Nexus1000V active * 2 0 Virtual Supervisor Module Nexus1000V ha-standby Mod Sw Hw --- ------------------ ------------------------------------------------ 1 4.2(1)SV1(5.1) 0.0 2 4.2(1)SV1(5.1) 0.0 Mod MAC-Address(es) Serial-Num --- ------------------------------------------------- ---------- 1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA 2 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA Mod Server-IP Server-UUID Server-Name --- --------------- -------------------------------- -------------------- 1 10.29.172.52 NA NA 2 10.29.172.52 NA NA * this terminal session J05-L3-VSM# show svs connections connection vcenter: ip address: 10.29.172.53 remote port: 80 protocol: vmware-vim https certificate: default datacenter name: Building2 admin: max-ports: 8192 DVS uuid: 89 96 15 50 e3 87 90 74-02 09 f7 3c 56 3b 2a fc config status: Enabled operational status: Connected sync status: Complete version: VMware vCenter Server 5.0.0 build-455964 vc-uuid: EEE20C26-2282-4C2B-8DF4-CFEC7EDA7E25

Configuring  Port-­‐Profiles  Once the VSM has been installed, the next task is to configure the port-profiles. Before you add the ESX/ESXi servers as VEMs, you must create the port-profiles. There are two types of port-profiles, type Ethernet and type vEthernet. Use the following sections as a guide in creating them.

Type  Ethernet  Port-­‐Profiles  The uplink port-profile will need to allow all of the VLANs for the environment. The other requirements are to configure the appropriate system VLANs and to configure the channel-group for the virtual port-channel for the VEMs. Before you configure the uplink port-profile, you must create the VLANs for the VSM. VLAN 1 is created by default. The following shows the configuration for creating the additional VLANs:

J05-UCSB-N1KV# configure terminal Enter configuration commands, one per line. End with CNTL/Z. J05-UCSB-N1KV(config)# vlan 51 J05-UCSB-N1KV(config-vlan)# name vMotion J05-UCSB-N1KV(config-vlan)# vlan 52 J05-UCSB-N1KV(config-vlan)# name Web J05-UCSB-N1KV(config-vlan)# vlan 53 J05-UCSB-N1KV(config-vlan)# name Database J05-UCSB-N1KV(config-vlan)# vlan 172 J05-UCSB-N1KV(config-vlan)# name Management J05-UCSB-N1KV(config-vlan)# show vlan VLAN Name Status Ports ---- ------------------- --------- ------------------------------- 1 default active 51 vMotion active 52 Web active 53 Database active 172 Management active VLAN Type ---- ----- 1 enet 51 enet 52 enet 53 enet 172 enet Remote SPAN VLANs ----------------------------------------------------------------- Primary Secondary Type Ports ------- --------- --------------- ----------------------------

With the VLANs created, here’s how to create the uplink port-profile:

J05-UCSB-N1KV# configure terminal Enter configuration commands, one per line. End with CNTL/Z. J05-UCSB-N1KV(config)# port-profile type ethernet system-uplink J05-UCSB-N1KV(config-port-prof)# vmware port-group J05-UCSB-N1KV(config-port-prof)# switchport mode trunk J05-UCSB-N1KV(config-port-prof)# switchport trunk allowed vlan 1, 51-53, 172 J05-UCSB-N1KV(config-port-prof)# no shutdown J05-UCSB-N1KV(config-port-prof)# system vlan 1, 172 J05-UCSB-N1KV(config-port-prof)# channel-group auto mode active J05-UCSB-N1KV(config-port-prof)# state enabled J05-UCSB-N1KV(config-port-prof)# show running-config port-profile system-uplink

!Command: show running-config port-profile system-uplink !Time: Wed Apr 11 13:01:13 2012 version 4.2(1)SV1(5.1) port-profile type ethernet system-uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1,51-53,172 channel-group auto mode active no shutdown system vlan 1,172 state enabled

Note: For layer 3 mode, you are required to set the management VLAN to be a system VLAN within the uplink port-profile.

Type  vEthernet  Port-­‐Profiles  Once you’ve created the uplink port-profile, it’s time to create the port-profiles used by virtual machines and VMkernels. These profiles are of type vEthernet, which is the default type. With layer 3 communication between the VSM and VEM, a port-profile of type vEthernet is needed that is capable to do this layer 3 communication. During the installer application procedure, this port-profile was already created. The name of this port-profile is n1kv-L3. The configuration output is shown below.

!Command: show running-config port-profile n1kv-L3 !Time: Wed Apr 11 13:02:27 2012 version 4.2(1)SV1(5.1) port-profile type vethernet n1kv-L3 capability l3control vmware port-group switchport mode access switchport access vlan 172 no shutdown system vlan 172 state enabled

Note: This port-profile has the entry capability l3control and is configured as a system vlan. The following shows the port-profiles of type vEthernet for the rest of the environment.

J05-UCSB-N1KV# configure terminal Enter configuration commands, one per line. End with CNTL/Z. J05-UCSB-N1KV(config)# port-profile vmotion J05-UCSB-N1KV(config-port-prof)# vmware port-group J05-UCSB-N1KV(config-port-prof)# switchport mode access J05-UCSB-N1KV(config-port-prof)# switchport access vlan 51 J05-UCSB-N1KV(config-port-prof)# no shutdown J05-UCSB-N1KV(config-port-prof)# state enabled J05-UCSB-N1KV(config-port-prof)# exit J05-UCSB-N1KV(config)# port-profile Web J05-UCSB-N1KV(config-port-prof)# vmware port-group

J05-UCSB-N1KV(config-port-prof)# switchport mode access J05-UCSB-N1KV(config-port-prof)# switchport access vlan 52 J05-UCSB-N1KV(config-port-prof)# no shutdown J05-UCSB-N1KV(config-port-prof)# state enabled J05-UCSB-N1KV(config-port-prof)# exit J05-UCSB-N1KV(config)# port-profile Database J05-UCSB-N1KV(config-port-prof)# vmware port-group J05-UCSB-N1KV(config-port-prof)# switchport mode access J05-UCSB-N1KV(config-port-prof)# switchport access vlan 53 J05-UCSB-N1KV(config-port-prof)# no shutdown J05-UCSB-N1KV(config-port-prof)# state enabled J05-UCSB-N1KV(config-port-prof)# exit J05-UCSB-N1KV(config)# port-profile vsm-control-packet J05-UCSB-N1KV(config-port-prof)# vmware port-group J05-UCSB-N1KV(config-port-prof)# switchport mode access J05-UCSB-N1KV(config-port-prof)# switchport access vlan 1 J05-UCSB-N1KV(config-port-prof)# system vlan 1 J05-UCSB-N1KV(config-port-prof)# no shutdown J05-UCSB-N1KV(config-port-prof)# state enabled J05-UCSB-N1KV(config-port-prof)# exit J05-UCSB-N1KV(config)# port-profile vsm-management J05-UCSB-N1KV(config-port-prof)# vmware port-group J05-UCSB-N1KV(config-port-prof)# switchport mode access J05-UCSB-N1KV(config-port-prof)# switchport access vlan 172 J05-UCSB-N1KV(config-port-prof)# system vlan 172 J05-UCSB-N1KV(config-port-prof)# no shutdown J05-UCSB-N1KV(config-port-prof)# state enabled J05-UCSB-N1KV(config-port-prof)# exit

Once all the port-profiles created, verify in vSphere that you can see them through vCenter. The window in Figure 21 verifies that the port-profiles have been synched to vCenter. Figure 21 Verifying the Port-Profiles

Adding  a  Server  as  a  VEM  When you add a VEM, there are two methods of installing the VEM binaries onto the ESX/ESXi servers: manually or through the VMware Update Manager (VUM). In our example, the VUM is installed and will be used. In this process, both the primary and secondary VSM will be migrated behind the VEM. In the procedure for adding the VEM, all the VMkernels will be migrated to the Cisco Nexus 1000V Series as well.

Adding  a  VEM,  Migrating  VMkernels  and  VSM  behind  the  VEM  The server 10.29.172.171 is hosting the primary VSM and will be the first server to be added to the Cisco Nexus 1000V. Note: The secondary VSM has already been migrated to the other server (10.29.172.172). From the Networking view (Figure 22), select the Nexus 1000V virtual switch (J05-L3-VSM) and click the Hosts tab. To add a host to this distributed virtual switch, right-click and select Add Host… or press Ctrl+H. Figure 22 Adding a Server as a VEM: Screen 1

The window shown in Figure 23 provides a list of all the servers. We will select the VMNICs for server 10.29.172.171 to be used by the Cisco Nexus 1000V. Once the checkbox is selected, you must select the Uplink port-group for those interfaces, which correlates to the uplink port-profile that was created in the previous section. Click the drop-down box, select system-uplink for both interfaces, as shown in Figure 23, and click Next. Figure 23 Adding a Server as a VEM: Screen 2

The next window lists the VMkernels on this server and provides the option to migrate the VMkernels over to the Cisco Nexus 1000V. Since the port-profiles have already been created, select the appropriate port-profiles for the listed VMkernels as shown in Figure 24. Then click Next.

Figure 24 Adding a Server as a VEM: Screen 3

The next window (Figure 25) lists the virtual machines that reside on this server. Since this server has only the primary VSM, click the checkbox called Migrate virtual machine networking and expand the server list to see the virtual machines. With the primary VSM network adapters, go to the Destination port group and select the appropriate port-profiles, as shown in Figure 25. Then click Next. Figure 25 Adding a Server as a VEM: Screen 4

Click Finished to complete the adding of the server. The VEM binaries will now be installed onto the server by VUM, and the server will be shown as another module in the VSM. The vCenter server will also see that the server has been added. The VMkernels and virtual Ethernet interfaces for the primary VSM will be added as well. Use the show commands to see the result shown in Figure 26. Figure 26 Adding a Server as a VEM: Screen 5

Note: Repeat this step for the rest of the servers to add them as VEMs. To verify that the VEM has been seen by the VSM, execute the following command:

J05-L3-VSM# show module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 1 0 Virtual Supervisor Module Nexus1000V active * 2 0 Virtual Supervisor Module Nexus1000V ha-standby 3 248 Virtual Ethernet Module NA ok 4 248 Virtual Ethernet Module NA ok Mod Sw Hw --- ------------------ ------------------------------------------------ 1 4.2(1)SV1(5.1) 0.0 2 4.2(1)SV1(5.1) 0.0 3 4.2(1)SV1(5.1) VMware ESXi 5.0.0 Releasebuild-469512 (3.0) 4 4.2(1)SV1(5.1) VMware ESXi 5.0.0 Releasebuild-469512 (3.0) Mod MAC-Address(es) Serial-Num --- ------------------------------------------------ ------------- 1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA 2 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA 3 02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA 4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA Mod Server-IP Server-UUID Server-Name --- --------------- ------------------------------------------------ -------------------- 1 10.29.172.52 NA NA 2 10.29.172.52 NA NA

3 10.29.172.171 6591e4a5-94d1-11df-a263-f866f222d9d8 10.29.172.171 4 10.29.172.172 fe4e5bb2-3243-11df-b443-68efbdf62444 10.29.172.172 * this terminal session J05-L3-VSM# show interface virtual --------------------------------------------------------------------------------------------- Port Adapter Owner Mod Host --------------------------------------------------------------------------------------------- Veth1 vmk0 VMware VMkernel 3 10.29.172.171 Veth2 vmk1 VMware VMkernel 3 10.29.172.171 Veth3 Net Adapter 1 J05-L3-VSM-1 3 10.29.172.171 Veth4 Net Adapter 2 J05-L3-VSM-1 3 10.29.172.171 Veth5 Net Adapter 3 J05-L3-VSM-1 3 10.29.172.171 Veth6 vmk0 VMware VMkernel 4 10.29.172.172 Veth7 vmk1 VMware VMkernel 4 10.29.172.172 Veth8 Net Adapter 1 J05-L3-VSM-2 4 10.29.172.172 Veth9 Net Adapter 2 J05-L3-VSM-2 4 10.29.172.172 Veth10 Net Adapter 3 J05-L3-VSM-2 4 10.29.172.172

Figure 27 List of VEMs from vCenter

Summary This completes the cheat sheet for the Nexus 1000V Configuration in Layer 3 mode. Other advance features can now be added and configured, such as QoS, ACL, DHCP Snooping, etc.