Nexus 1000v

www.silantia.com1

Nexus 1000v

VLANs Port Channels Port Profiles QoS Traffic Flow Multicast IGMP

Snooping Network Monitoring Troubleshooting

www.silantia.com2

Virtual Networking concept

Each ESXi host has mainly two types of network adaptor VmKernel port – Handles traffic for ESXi services like vMotion,

iSCSI, NFS and N1Kv Control. Virtual machine port – Handles traffic for virtual machines.

Vmware ESXi has two types of virtual switches. Standard vSwitch: managed as individual entity and cannot be

centrally configured Distributed virtual switch (DVS) : Managed as single entity

across multiple host and configured centrally. Each switch can have its own set of uplinks which

connects ESXi host to external world. All virtual machines and vmkernal ports connects to

vswitch and/or DVS.

www.silantia.com3

Virtual Networking concept

vSwitch/DVS are dot1q capable switchs which can switch traffic between virtual machine or Vmk ports.

vSwitch/DVS DOES not run spanning tree. vSwitch/DVS can have multiple VLANs and none of them

could do L3 routing. vSwitch/DVS looks up each frame’s destination MAC

when it arrives. It builds mac address table for local VMs.

If destination mac address is not present in MAC address table then traffic is sent to one of the uplink.

Traffic between two VMs gets pinned to uplink and switched via physical switch and goes to destination ESXi host where VM resides and vSwitch/DVS on that host switches that frame to desired VM.

www.silantia.com4

Introduction to Nexus 1000v

First CLI based virtual distributed switch. It’s a Cisco Nexus switch. It has two components

VSM ,Virtual Supervisor Module : VM which controls VEM VEM Virtual ethernet module : Software agent / module

running inside ESXi host. All components in a single vSphere cluster operate

logically as a single virtualised access switch

www.silantia.com5


www.silantia.com6


Protocols such as CDP, LACP, SNMP, IGMP operate as a single switch, These control protocols are run on the VSM and carried over the “packet VLAN” to the VEMs.

Because the VSM and VEM are not physically connected, the VSM (supervisor) must program the VEM (linecards) over a network. It does this by using Control and Packet network.

There are two ways to extend communication between VSM and VEM: Over Layer 2 network using Control and Packet VLANs called L2

mode Over Layer 3 network using Layer 3 Control Capability called L3

mode VEM has special VMK port which it uses to communicate with

VSM. L3 mode is more preferred because you can then put VEM

and VSM in separate subnet.

www.silantia.com7

More on L3 control mode

Requires an IP address be assigned to the VEM L3 uses UDP port 4785 for both source and destination Uses Mgmt or Control interface of the VSM VSM mgmt 0 is default interface for L3, it can also use

“control 0” interface Ties to control adapter of the VSM VM (Adapter 1) Control0 interface uses default VRF and mgmt0 uses

management VRF. Primary and secondary VSM still need to be L2 adjacent

www.silantia.com8

L3 VEM Deployment Requirements

VSM setup for L3 control Create VMK Interface on ESXi host Test ping from control0 or mgmt0 to VMK of ESXi host

Create uplink port-profile Create veth port-profile with “capability l3control”

Migrate host and VMK interface VEM gets installed via VUM, manual, or installer app Host will not show up until VMK interface is moved to port-

profile

www.silantia.com9

L3 Control mode configurationsvs-domain

domain id 50

svs mode L3 interface control0

interface control0

ip address 172.20.186.11/24

port-profile type vethernet N1K-Control

capability l3control

vmware port-group

switchport mode access

switchport access vlan 260

no shutdown

system vlan 260

description N1KV Control

state enabled

port-profile type ethernet NON-LACP-UPLINKS

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 1,100,260-261

channel-group auto mode on mac-pinning

no shutdown

system vlan 1,100,260-261

state enabled

www.silantia.com10


The ports that connects to VMs are called veth ports and ports that are connected to Physical network switches are called uplink ports.

Ports are never configured directly, instead it is configured with port-profile (a set of ports with similar properties such as VLAN ID, QoS, ACL, etc.)

These port profiles appear in vcenter as a port group. Server admin can then select a port group defined by network admin for a network adaptor while creating a VM.

Editing an enabled profile will cause config changes to propagate to all interfaces using that profile (unlike a static one-time smartport macro in catalyst switch)

Two types Type Ethernet used for physical NIC uplinks Type Vethernet used for VM network connectivity

www.silantia.com11

Example port-profiles.

port-profile type ethernet UPLINK

vmware port-group


switchport trunk native vlan 1

switchport trunk allowed vlan 1-3967,4048-4093

service-policy type queuing output UPLINK_QUEUING


no shutdown


state enabled

port-profile type vethernet Management

vmware port-group

switchport mode access


no shutdown

system vlan 1

description ESXi management

state enabled

www.silantia.com12

Nexus 1000v System Vlans

A system vlan should be used whenever need to pass traffic through a VEM even when it can’t connect to VSM.

A system vlan is any vlan needed for functionality in a corner case, that is, when the VSM is offline and the ESXi host with the VEM is rebooted.

A system vlan is used to config and bring up physical or veth ports BEFORE the VSM has established communication with VEM.

Examples are VLANs that are used for control, packet, ESXi Management and iSCSI.

You define these vlans as system vlans under corresponding port-profile and uplink port-profile.

www.silantia.com13

vCenter to VSM communication

VSM connects to vCenter using SSL connection using mgmt0 IP address as source.

VSM configures vCenter using its API VSM creates N1KV Port-Groups in vCenter VSM also stores opaque data in vCenter VSM pulls information from vCenter (DC, DVS, VM, …)

VSM continuously able to talk to vCenter alltime there is config changes is done.

VSM registers to vCenter using extension key plug-in Contains a public SSL Certificate Extension key of the VSM

www.silantia.com14

vCenter to VSM communication

N1KV-VSM# sh svs connections

connection vcenter:

ip address: 192.168.2.99

remote port: 80

protocol: vmware-vim https

certificate: default

datacenter name: CCIEDC

admin: n1kUser(user)

max-ports: 8192

DVS uuid: cd 26 09 50 e5 55 9e d3-1b 17 d4 9c 1c ae 4f 6e

config status: Enabled

operational status: Connected

sync status: Complete

version: VMware vCenter Server 5.1.0 build-947673

vc-uuid: EE1F2CA8-2DEE-43E4-826D-472D18E6BF89

www.silantia.com15

Nexus 1000v Port-channels

You can bundle uplinks into a port-channel. Uplinks from different ESXi hosts cannot be bundled into a port-channel.

Veth cannot be bundled into a port-channel. If Physical switch supports LACP then it can be used in

Nexus 1000v to dynamically form port-channel. If upstream switch does not support LACP then you will

need to use a concept called MAC pinning (more in later slide).

LACP offload feature: This feature allows LACP negotiation off loaded to VEM code i.e. VEM can do LACP negotiation with upstream switch without even connecting to VSM.

www.silantia.com16


Scenario 1 upstream switches supports LACP.

www.silantia.com17


Scenario I upstream switches supports LACP.port-profile type ethernet LACP_UPLINKS

vmware port-group




mtu 9000

channel-group auto mode active

no shutdown


state enabled

www.silantia.com18


Scenario 1I upstream switches DOES NOT support LACP.

www.silantia.com19


Scenario II upstream switches DOES NOT support LACP.

port-profile type ethernet NON-LACP-UPLINKS

vmware port-group




mtu 1500

service-policy type queuing output UPLINK_QUEUING



no shutdown


state enabled

www.silantia.com20


Scenario II upstream switches DOES NOT support LACP.

N1KV-VSM# sh port-channel summary

Flags: D - Down P - Up in port-channel (members)

I - Individual H - Hot-standby (LACP only)

s - Suspended r - Module-removed

S - Switched R - Routed

U - Up (port-channel)

--------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------

1 Po1(SU) Eth NONE Eth3/1(P) Eth3/2(P)

www.silantia.com21

Nexus 1000v and UCS

www.silantia.com22

Nexus 1000v and UCS

End host mode Enable CDP on Network control policy. Enable Host control full in QoS policy. No Fabric failover. Use one, two or four vNIC template for each

fabric. Use On Nexus 1000v channel-group auto

mode on pinning on uplink port-profile. L3 control mode. Use ESXi mgmt, Control and iSCSI VLANs as

System VLANs.

www.silantia.com23

Nexus 1000v and QoS

Nexus 1000v leverage the MQC qos-group capabilities to identify and define traffic in policy configuration. Define matching criteria via a class-map Associate action with each defined class via a

policy-map Apply policy to entire system or an interface via a

service-policy Nexus1000v: 64 classes (8 pre-defined) . Since Nexus 1000v is access layer switch so

Classification, Marking and Queuing is done.

www.silantia.com24

Nexus 1000v and QoS Classification Criteria:

CoS IP precedence DSCP Layer 2 to Layer 4 parameters ACL QoS group Discard class Protocol

Marking allowed: CoS IP precedence DSCP QoS group Discard class

Policing Conditions: Conforms to rate limits Exceeds rate limit maximum Violates rate limit Policing Types: Single rate (CIR) Dual rate (CIR and PIR) Color aware

www.silantia.com25

Nexus 1000v and QoS

Ingress QoS policies per interface: One type QoS One type queuing

Egress QoS policies per interface: One type QoS One type queuing

www.silantia.com26

Nexus 1000v and QoS Configuration

N1KV-VSM# sh class-map

Type queuing class-maps

========================

class-map type queuing match-any iSCSI

match protocol vmw_iscsi

class-map type queuing match-any vMotion

match protocol vmw_vmotion

class-map type queuing match-all Management

match protocol vmw_mgmt

match protocol n1k_mgmt

class-map type queuing match-all N1KV-Control-Packet

match protocol n1k_packet

match protocol n1k_control

www.silantia.com27


Type qos policy-maps

====================

policy-map type qos SET_COS2

class class-default

set cos 2


class class-default

set cos 4

Type queuing policy-maps

========================

policy-map type queuing UPLINK_QUEUING

class type queuing Management

bandwidth percent 5

class type queuing N1KV-Control-Packet

bandwidth percent 5

class type queuing iSCSI

bandwidth percent 30

class type queuing vMotion


www.silantia.com28


Type qos policy-maps

====================


class class-default

set cos 2


class class-default

set cos 4

Type queuing policy-maps

========================

policy-map type queuing UPLINK_QUEUING

class type queuing Management

bandwidth percent 5

class type queuing N1KV-Control-Packet

bandwidth percent 5

class type queuing iSCSI


class type queuing vMotion


www.silantia.com29


port-profile type ethernet NON-LACP-UPLINKSservice-policy type queuing output UPLINK_QUEUING

N1KV-VSM# show policy-map interface po1

Global statistics status : enabled

port-channel1

Service-policy (queuing) output: UPLINK_QUEUING

policy statistics status: enabled

Class-map (queuing): N1KV-Control-Packet (match-all)

Match: protocol n1k_packet

Match: protocol n1k_control

bandwidth percent 5

queue dropped pkts : 0

queue matched pkts : 0

queue inrate bytes ( Kbits/sec ) : 0

queue outrate bytes ( Kbits/sec ) : 0

www.silantia.com30

Nexus 1000v Network Monitoring

Each Veth port has interface statistics just like physical ports.

These statistics are preserved when attached VM moved from one host to another host.

Veth ports are brought up on demand it should never be configured individually.

Uplink physical ports also has interface statistics. Ethanalyzer can be used to capture control0 and

mgmt0 interfaces.ethanalyzer local sniff-interface control dump-pkt write bootflash:control0.pcap

Netflow version 5 and version 9 are supported.

www.silantia.com31

Nexus 1000v Netflow

Feature netflow

flow exporter NetflowExporter

destination 192.168.2.11 use-vrf management

source mgmt0

version 9

flow record Inbound

description Netflow record for inbound traffic

match ipv4 destination address

match ip protocol

match ip tos

match transport destination-port

collect transport tcp flags

collect counter bytes

collect counter packets

flow monitor NetflowMonitor

record Inbound

exporter NetflowExporter

port-profile type vethernet NON-NLB

ip flow monitor NetFlowMonitor input

www.silantia.com32

Nexus 1000v ERSPAN

Encapsulated Remote Switched Port Analyzer Nexus 1000v does not support destination erspan A veth port can be captured and sent accross IP network to

destination in to a GRE encapsulated packet where it can bet decapsulated and analysed.

Capability L3-control has to be specified for port-profile of ERSPAN source. This port profile is applied to vmk port of hypervisor and it is used as source of a GRE tunnel.

N1KV-VSM(config)# monitor session 1 type erspan-source

N1KV-VSM(config-erspan-src)# source ?

interface Configure interfaces

port-profile Port profile name

vlan Vlan type

N1KV-VSM(config-erspan-src)# destination ip 192.168.2.169

www.silantia.com33

Nexus 1000v IGMP snooping

Nexus 1000v is full IGMP snooping bridge but it cannot perform IGMP querier job.

ip igmp snooping ! Enables at global level

vlan 200

ip igmp snooping ! Enables per vlan IGMP snooping

ip igmp snooping explicit-tracking

ip igmp snooping mrouter interface ethernet 2/1

! vEths are not supported as router ports

ip igmp snooping static-group 230.0.0.1 interface vethernet 21

show ip igmp snooping vlan 200

www.silantia.com34

Nexus 1000v Layer 2 Security feature

Works exactly similar to Layer 2 security features in physical switches.

Security features requires Nexus 1000v Advanced License.

Layer 2 security is important in Virtual Desktop Infrastructure type of environment where each virtual machine is a user desktop.

Unmanaged VMs can bring down the whole Layer 2 network if it is not protected at VEM level.

www.silantia.com35

Nexus 1000v Port-security

MAC-to-Port Mapping Don't allow any MAC addresses other than those mapped

to pass traffic Static = Static MAC-to-Port Mapping Dynamic = Learn the MAC, map to the port, then don't

allow anyone ELSE Can also age this dynamic mapping out

Sticky = Same as dynamic, but store mapping in NVRAM Violations

Shutdown = Shuts port down. Simple, done. Restrict = Drops traffic from any Other MAC addys

Protect = Basically drops traffic from any other MACs like Restrict, but first it learns the MAC of the 1st violator and logs him (still drops his traffic too)

and doesn't learn any other violators' MACs

www.silantia.com36

Nexus 1000v Port-security

You can configure port security only on Layer 2 interfaces Details about port security and different types of

interfaces or ports are as follows: Access ports

You can configure port security on interfaces that you have configured as Layer 2 access ports

On an access port, port security applies only to the access VLAN Trunk ports

You can configure port security on interfaces that you have configured as Layer 2 trunk ports

The device allows VLAN maximums only for VLANs associated with the trunk port

SPAN ports You can configure port security on SPAN source ports but not on

SPAN destination ports

www.silantia.com37

Nexus 1000v DHCP snooping

DHCP snooping functions like a firewall between untrusted hosts and trusted DHCP servers by doing the following: Validates DHCP messages received from untrusted sources and

filters out invalid response messages from DHCP servers. Builds and maintains the DHCP snooping binding database, which

contains information about untrusted hosts with leased IP addresses.

Uses the DHCP snooping binding database to validate subsequent requests from untrusted hosts.

Dynamic ARP Inspection (DAI) and IP Source Guard also use information stored in the DHCP snooping binding database.

When you enable DHCP snooping, by default, all vEthernet (vEth) ports are untrusted and all Ethernet ports (uplinks), port channels, special vEth ports are trusted.

www.silantia.com38


DHCP operations are categorized into four basic phases:IP Discovery IP Lease Offer IP Request IP Lease Acknowledgement

Only DHCP messages that come from a server that is connected to a trusted port are accepted.

Any DHCP message on UDP port 68 that is data from the server to the client that is received on an untrusted port is dropped. Nexus 1000v VEM builds and maintains the DHCP snooping binding database, which contains information about clients with leased IP addresses.

Uses the DHCP snooping binding database to validate subsequent requests from clients.

www.silantia.com39


Configuration Enable the DHCP feature.

feature dhcp Enable DHCP snooping globally.

ip dhcp snooping Enable DHCP snooping on at least one VLAN.By default,

DHCP snooping is disabled on all VLANs. Ensure that the ip dhcp snooping vlan vlan-list DHCP server is connected to the device using a trusted

interface.N1KV-VSM(config)# port-profile profilename

N1KV-VSM(config-if)# ip dhcp snooping trust Configuring the Rate Limit for DHCP PacketsN1KV-VSM(config-if)#[no] ip dhcp snooping limit rate rate

www.silantia.com40


Configuration Error disable detection and recovery errdisable detect cause dhcp-rate-limit

Enables DHCP error-disabled detection.errdisable recovery cause dhcp-rate-limit Enables DHCP error-disabled detection.errdisable recovery interval time interval Sets the DHCP error-disabled recovery interval, where time

interval is the number of seconds from 30 to 65535.

www.silantia.com41


VerificationN1KV-VSM# show ip dhcp snooping

Switch DHCP snooping is enabled

DHCP snooping is configured on the following VLANs:

100,200,250-252

DHCP snooping is operational on the following VLANs:

100,200,250-252

Insertion of Option 82 is disabled

Verification of MAC address is enabled

DHCP snooping trust is configured on the following interfaces:

Interface Trusted Pkt Limit

------------ ------- ---------

Vethernet1 No Unlimited


Vethernet3 Yes 15



www.silantia.com42


VerificationN1KV-VSM# show ip dhcp snooping statistics

Packets processed 0

Packets forwarded 0

Total packets dropped 0

Packets dropped from untrusted ports 0

Packets dropped due to MAC address check failure 0

Packets dropped due to Option 82 insertion failure 0

Packets dropped due to o/p intf unknown 0

Packets dropped which were unknown 0

Packets dropped due to service dhcp not enabled 0

Packets dropped due to no binding entry 0

Packets dropped due to interface error/no interface 0

Packets dropped due to max hops exceeded 0

www.silantia.com43

Nexus 1000v Dynamic ARP inspection

DAI ensures that only valid ARP requests and responses are relayed by intercepting all ARP requests and responses on untrusted ports and verifying that each of these intercepted packets has a valid IP-to-MAC address binding before updating the local ARP cache or before forwarding the packet to the appropriate destination.

DAI depends on the entries in the DHCP snooping binding database to verify IP-to-MAC address bindings in incoming ARP requests and ARP responses.

DAI is supported on vEthernet interfaces and private VLAN ports

www.silantia.com44


Configuration:N1KV-VSM(config)# ip arp inspection vlan #

N1KV-VSM(config)# port-profile profilename

N1KV-VSM(config-port-profile)# ip arp inspection trust

Verificationswitch# show ip arp inspection interfaces vethernet 3 Interface Trust State Pkt Limit Burst Interval

------------- ----------- --------- ----------------

Vethernet9 Untrusted 30 5

www.silantia.com45


Rate limiting ip arp inspection limit {rate pps [burst interval l bint] | none}

Configures the specified ARP inspection limit on the interface or the port profile as follows. rate—Specifies that allowable values are between 1

and 2048 packets per second (pps). The untrusted interface default is15 packets per second. The trusted interface default is15 packets per second.

burst interval—Specifies that allowable values are between 1 and 15 seconds (the default is 5 seconds).

none—Specifies an unlimited number of packets per second.

www.silantia.com46


Can enable additional validation checks ip arp inspection validate ? src-mac: Checks the source MAC address in the Ethernet header against the

sender MAC address in the ARP body for ARP requests and responses

dst-mac: Checks the destination MAC address in the Ethernet header against the target MAC address in the ARP body for ARP responses

ip: Checks the ARP body for invalid and unexpected IP addresses. Addresses include 0.0.0.0, 255.255.255.255, and all IP multicast addresses

arp access-list UNK-SW permit ip host 10.0.0.1 mac host 0000.0000.0001 ip arp inspection filter UNK-SW vlan 10

Error disable Port may go into error disable when ARP inspection is violated.

N1KV-VSM(config)# errdisable detect cause arp-inspection

N1KV-VSM(config)# errdisable recovery cause arp-inspection You can shut no shut port or configure error disable recovery to recover automatically.

www.silantia.com47

Nexus 1000v IP Source Guard

IP SG is a per-interface traffic filter that permits IP traffic only when the IP address and MAC address of each packet matches one of two sources of IP and MAC address bindings Entries in DHCP snooping binding table. Static IP source entries that you configure.

You can enable IP Source Guard on Layer 2 interfaces that are not trusted by DHCP snooping.

When you initially enable IP Source Guard, all inbound IP traffic on the interface is blocked except for the following: DHCP packets, which DHCP snooping inspects and then

forwards or drops, depending upon the results of inspecting the packet.

IP traffic from a source whose static IP entries are configured in the Cisco Nexus 1000V.

www.silantia.com48

Nexus 1000v IP Source Guard

Configuration: (can be done under port-profile)N1KV-VSM(config)# interface vethernet 31

N1KV-VSM(config-if)# ip verify source dhcp-snooping-vlan

Verification: switch (config-if)# show ip verify source interface vethernet 3

Filter Mode(for static bindings): IP-MAC IP source guard is enabled on this interface.

Interface Filter-mode IP-address Mac-address Vlan

---------- ----------- ---------- ----------- ----

Vethernet3 active 1.182.56.137 00:50:56:82:56:3e 1053

Adding Static entry in for IP SG.N1KV-VSM(config)# ip source binding 10.5.22.17 001f.28bd.0013 vlan 100 interface vethernet 3

www.silantia.com49

Nexus 1000v DAI and IPSG

www.silantia.com50

Nexus 1000v ACL

Two types of ACLs are supported in Nexus 1000v IP ACL- Applied only to IP traffic MAC ACL – Applied only to Non IP traffic

Order of ACL application Ingress port ACL Egress port ACL

MAC ACL supports following additional filtering options.

Layer 3 protocolVLAN ID Class of Service (CoS)

www.silantia.com51

Nexus 1000v IP ACL

IP ACL supports following additional filtering options. Layer 4 protocol TCP and UDP ports ICMP types and codes IGMP types Precedence level Differentiated Services Code Point (DSCP) value TCP packets with the ACK, FIN, PSH, RST, SYN, or URG bit

set All ACL are configured via CLI on VSM and when ACLs

are applied to port-profile or veth/ethernet port it will be processed at VEM level.

www.silantia.com52

Nexus 1000v IP ACL

Configuration example:ip access-list DENY_OSPF

10 deny ip any 224.0.0.5/32

20 deny ip any 224.0.0.6/32

30 permit ip any any

ip access-list DENY_TELNET

10 deny tcp any 150.10.2.1/32 eq telnet

20 permit ip any any

port-profile type veth SERVERFARM1

ip access-group DENY_TELNET in

www.silantia.com53

Nexus 1000v Private VLANs

Private VLANs partition a regular VLAN domain into subdomains and can have multiple VLAN pairs.

All VLAN pairs in a private VLAN share the same Primary VLAN. The secondary VLAN ID differentiates one subdomain from another.

All members in the private VLAN share a common address space, which is allocated to the primary VLAN.

Private VLANs can span multiple switches. A trunk port carries the primary VLAN and secondary VLANs to a neighboring switch. (uplink ports in case of Nexus 1000v)

www.silantia.com54

Nexus 1000v Private Vlans

Enable private vlan and configure primary and secondary vlans

feature private-valnvlan 153private-vlan primaryprivate-vlan association 154-155vlan 154private-vlan communityvlan 155private-vlan isolated

www.silantia.com55


! Private vlan configured on port-profileport-profile type vethernet pv154vmware port-groupswitchport mode private-vlan hostswitchport private-vlan host-association 153 154no shutdownstate enabled

! You can configure private vlan on Veth port itself.port-profile type vethernet pv155vmware port-groupswitchport mode private-vlan hostswitchport private-vlan host-association 153 155no shutdownstate enabled!

www.silantia.com56


Create uplink port-profile carrying private vlans.

port-profile type ethernet pcpvtrunkvmware port-groupswitchport mode private-vlan trunk promiscuousswitchport private-vlan mapping trunk 153 154-155switchport private-vlan trunk allowed vlan 153-155channel-group auto mode on mac-pinningno shutdownstate enabled

Technology

Nexus 1000v