Click here to load reader

Deploying and Troubleshooting the Nexus 1000v …d2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKVIR-3013.pdfDeploying and Troubleshooting the Nexus 1000v Virtual Switch on vSphere BRKVIR-3013

  • View
    241

  • Download
    8

Embed Size (px)

Text of Deploying and Troubleshooting the Nexus 1000v...

  • Deploying and Troubleshooting the Nexus 1000v Virtual Switch on vSphere

    BRKVIR-3013

    Matthew Wronkowski Technical Leader Virtualization Services

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Agenda

    Current N1K Releases and New Features

    Licensing

    Virtual Supervisor Module (VSM) & VEM

    VSM High Availability

    Upgrades

    Port-Profiles & Port Channels

    VXLAN

    Cisco Cloud Services Platform / Nexus1x10

    3

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Nexus 1000V Virtual Switch | Build & Price

    4

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Virtual Networking and Cloud Network Services

    Nexus 1000V

    Distributed Switch

    NX-OS consistency

    VSG

    VM-level controls

    Zone-based FW

    ASA 1000V

    Edge firewall, VPN

    Protocol Inspection

    vWAAS

    WAN optimization

    Application traffic

    WAN

    Router

    Servers

    Tenant A ASA

    1000V

    Cloud

    Firewall

    Nexus 1000V Physical

    Infrastructure

    Virtualized/Cloud Data Center

    vWAAS

    Cisco

    Virtual

    Security

    Gateway

    Switches

    Cloud Network Services

    Citrix

    NetScaler

    VPX

    Imperva

    SecureSphere

    WAF Cloud

    Services

    Router

    1000V

    Zone A

    Zone B

    vPath VXLAN

    Multi-Hypervisor (VMware, Microsoft, Ubuntu, RedHat*)

    Network

    Analysis

    Module

    (vNAM)

    vNAM

    App Visibility (L2-L7)

    CSR 1k

    WAN GW

    Routing & VPN

    Ecosystem

    Citrix NetScaler

    Imperva Web FW

    5

  • Name a feature we will not implement on Nexus 1000V.

    Saravan Rajendran, Cisco CNSG VP

    6

  • Current Releases and New Features

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Current Nexus 1000V Releases

    ESX 5.2(1)SV3(1.1)*

    256 VEMs, 12K vEth count

    VXLAN 2.0 (BGP)

    N1K Management Center

    ESX 4.2(1)SV2(2.2)

    Dynamic Fabric Automation Leaf

    VDP VSI Discovery Protocol

    Universal Licensing

    ESX - 4.2(1)SV2(2.1a)

    Scalability Release 128 VEMs

    VXLAN 1.5, VXLAN GW

    Geographically Separated VSMs

    Removed ESX 4.1 support

    Hyper-V 5.2(1)SM1(5.2a)

    SCVMM 2012 SP1 & R2

    Windows Server 2012 & R2

    VSG VM and Custom Attributes

    Universal Licensing

    InterCloud 5.2(1)IC1(1.2)

    Simplified Platform Image

    Local License Server or Cisco PNSC

    Ubuntu KVM / OpenStack

    Initial Release

    8

    *Next Release

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Evolution of VXLAN to version 1.5

    Unicast mode

    Simplifies VXLAN deployment

    Reduces network dependency (no multicast)

    Easier troubleshooting

    Flood directly to VXLAN Tunnel End Points (VTEP)

    Unicast Mac-address Distribution Mode

    Flooding is eliminated

    VSM learns all MACs and programs mappings to VEMs

    Faster response time

    Will not support VXLAN veth trunking(multi-mac)

    Requires static MACs (wont work with MS NLB)

    9

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    vTracker Feature

    Provides intuitive virtualization perspective to the network-admin

    Pulls data from vCenter and VEM

    Gives cloud view of connected objects

    Enabled with feature vtracker

    There are 5 view options

    module-view

    upstream-view

    vlan-view

    vm-view

    vmotion-view

    10

    SV2# show vtracker vm-view info vm win3

    Module 5:

    VM Name: win3

    Guest Os: Microsoft Windows Server 2003

    Standard (32-bit)

    Power State: Powered On

    VM Uuid: 423ca4df-26d0-50c1-d531-1a49b3a83aed

    Virtual CPU Allocated: 1

    CPU Usage: 0 %

    Memory Allocated: 1024 MB

    Memory Usage: 7 %

    VM FT State: Unknown

    Tools Running status: Running

    Tools Version status: current

    Data Store: datastore1 (2)

    VM Uptime: 25 days 3 hours 56 minutes 15s

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Nexus 1000V Manager Installation Screenshot

    Zero CLI full GUI interface

    Auto Host Selection

    Deploy Redundant VSMs

    Best Practices Auto-Implemented

    Automated prompts with suggestion for alternatives

    Customize Installation for Advanced Users

    *Available Summer 2014

    Install / Migrate / Upgrade / Monitor

    11

  • Licensing Info

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Licensing Essential Edition (No Expiration)

    Default mode for New Installs

    All features except

    Cisco TrustSec (CTS)

    DHCP Snooping

    IP Source Guard / Dynamic ARP Inspection

    Virtual Security Gateway (VSG)

    VXLAN Gateway

    128 modules with 4096 virtual ports

    Support Options

    Pay Nothing support is through the communities site off cisco.com https://communities.cisco.com/community/technology/datacenter/nexus1000v

    Pay for service contract

    13

    https://communities.cisco.com/community/technology/datacenter/nexus1000v

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Licensing Advanced Edition

    For customers that want more security features

    Customers with existing licenses will be considered Advanced

    Upgrade process will migrate VSM to Advanced Edition

    Required for VXLAN Gateway and VSG

    Licensed customers can get Virtual Security Gateway(VSG) for free

    Cisco Account Team can submit request

    VSG will no longer be sold separately

    256 modules with 12k virtual ports (SV3)*

    60-day Trial after which Advanced FeatureSet is disabled

    14

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Universal Licensing

    A common license is shared for both N1k & VSG.

    Cross Hypervisor portability.

    The license name is NEXUS1000V_LAN_SERVICES_PKG.

    Following upgrade, request a new Permanent license within 60 days.

    15

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Licensing New Commands

    Display Current Edition switch# show switch edition

    To switch between Essential or Advanced switch(config)# svs switch edition [essential | advanced]

    VEM Licenses are Sticky Removed & Offline VEMs hold a license switch# show module vem license-info

    Licenses are Sticky

    Mod Socket Count License Usage License Version License Status

    --- ------------ ------------- --------------- --------------

    3 2 2 1.0 licensed

    VEM license transfer to pool: switch(config)# svs license transfer src-vem license_pool

    16

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Licensing Overdraft Licenses

    Extra licenses to use in temporary situations

    16 extra sockets

    Sometimes more depending on number of licenses youve purchased

    Can only be used after a valid license is installed

    No penalty

    Full TAC Support for Overdraft Modules

    17

    SV2# show license usage NEXUS1000V_LAN_SERVICES_PKG

    ----------------------------------------

    Feature Usage Info

    ----------------------------------------

    Installed Licenses : 16

    Default Eval Licenses : 0

    Max Overdraft Licenses : 16

  • Virtual Supervisor Module Deployment and Troubleshooting

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Nexus 1000V Architecture

    Hypervisor Hypervisor Hypervisor

    VEM-N VEM-1 VEM-2

    VSM: Virtual Supervisor Module

    VEM: Virtual Ethernet Module Server

    Admin

    NX-OS

    Data Plane

    VSM-1 (active)

    VSM-2 (standby)

    Virtual Appliance

    NX-OS

    Control Plane Network

    Admin

    Modular Switch

    Linecard-N

    Supervisor-1 (Active)

    Supervisor-2 (StandBy)

    Linecard-1

    Linecard-2

    Ba

    ck P

    lan

    e

    19

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Virtual Supervisor Module (VSM)

    VSM is a Virtual Machine

    On ESXi, Hyper-V, Ubuntu KVM / OpenStack

    On Nexus 1x10 / Cloud Services Platform

    Control plane for the Nexus 1000V solution

    VEM packet forwarding not impacted by reloads

    Responsible for

    Programming and Managing Virtual Ethernet Modules (VEM)

    Communicating with Management Applications VMware vCenter, SCVMM, Horizon Dashboard

    1 VSM HA pair can manage 128 VEMs

    Coexist with VMware vSwitch, vDS, Microsoft Logical, Native Switches

    20

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Nexus 1000V VSM Interfaces

    Control

    L2/L3 VEM (AIPC)

    VSM-VEM Heartbeats (L2/L3)

    VSM-VSM Synchronization (L2)

    VSM-VSM HA Heartbeats (L2/L3)

    Packet

    CDP, IGMP, NetFlow, SNMP

    L3 Mode

    Collapsed Ctrl, Pkt into mgmt0

    VSM-VEM flow from mgmt0

    Dedicated Control: svs mode L3 interface [control | mgmt0]

    Management

    SSH console access

    SNMP, HTTP

    vCenter Communication

    HA Heartbeat Backup

    Interface Order is always the same!

    VSM-P eth0: control

    eth1: mgmt0

    eth2: packet

    21

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Deployment Scenarios

    Supports the VSM on a VEM

    Supports the VSM on any hypervisor native, logical, or distributed switch

    Supports the VSM on any supported hypervisor (ESXi/Hyper-V/N1110)

    Keep VSMs on different physical hosts

    Use anti-affinity rules

    Storage wise we dont care.

    VSM can be hosted on network storage

    22

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Stretched Nexus 1000V Model

    VSMs and VEMs spread across Datacenters

    VSMs can be split across DCs

    Requires L2 connectivity across DCI

    10ms latency across DCI

    Not supported with Hyper-V

    Supported in a future release

    23

    VSM

    VSM

    hypervisor

    VEM-1

    VM VM VM

    Local DC

    hypervisor

    VEM-2

    hypervisor

    VEM-4

    VM VM VM

    hypervisor

    VEM-3

    VM VM VM

    Remote DC

    VM VM

    DCI

    VM

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Control Modes

    L3 Mode

    L3 is the recommended & default Easier to troubleshoot

    Flexible

    Requires an IP address be assigned to the VEM

    Uses UDP4785 for both source and destination

    Sourced from mgmt0 by default

    L2 mode

    Requires L2 connectivity through control0 interface to all VEM modules

    L2 still supported on ESX

    Not supported with Hyper-V or KVM

    24

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM L3 Configuration and Planning

    Two options for the L3 control interface

    mgmt0 (default)

    control0

    Use Control0 to separate control and management traffic

    Mgmt and Control use different VRF

    mgmt0 uses VRF management

    control0 uses VRF default

    Primary and Secondary VSM still need to be L2 adjacent!

    Test with mping broadcast command. 0x201 is control between VSMs

    25

    # mping broadcast

    64 bytes from node 0x0201 (msg id = 0x030b1e 1) (time=0 sec, 1510 usec)

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Connectivity to VMware vCenter

    VSM connects to vCenter using SSL connection

    VC Extension contains the SSL cert

    Unique extension ID for the VSM

    Ability to generate own certificates

    VSM talks to vCenter using its API

    We push and pull data to/from vCenter

    VSMs get tied to a VMware Datacenter

    Multiple VSMs tied to same DC is allowed

    VSM can manage across clusters but not datacenters Can get confusing

    26

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Connectivity Errors - ESXi

    If you get Extension key was not registered before its use

    Re-register the Extension Key with VMware vCenter

    If you get Connection refused. connect failed in tcp_connect()

    Ping vCenter IP from VSM CLI

    VMware admin could have changed the http port

    API communication is through port 80 with VMware vCenter

    Find new port and change it on VSM

    27

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM and vMotion/Live Migration

    Manual vMotion/Live Migration is supported

    VMware DRS is NOT recommended for Primary & Secondary VSMs

    Aggressive settings could lead to excessive VSM-VEM heartbeat packet drops

    Best practice to keep Primary and Secondary VSM outside DRS control

    Use anti-affinity rules where possible

    28

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Backing up the VSM

    A running-config is not enough to restore

    VSM on ESXi

    Clone to a template

    You can restore from a template and saved-config

    Must be powered down

    VSM on Nexus 1x10

    Export a VSM to a file

    Import the saved VSM to restore

    VSM on ESXi Snapshots

    Not officially supported

    I/O latency cost associated with expanding the differential file

    29

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Best Practices - Summary

    L3 control is the preferred method

    Use mgmt0 for control traffic

    Primary and Standby VSM in same L2 domain!!!

    Required even if VSMs are split between datacenters

    VSM on VEM is supported

    10ms Latency between components: VSM-VSM, VSM-VEM

    10ms even for VSMs split between datacenters

    For VEMs at branch locations 100ms

    Backup your config!!!

    30

  • Nexus 1000V High Availability

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Redundancy Manager

    HA had to evolve to support split datacenter VSMs

    New Redundancy Manager process polls:

    VEM Manager polls for number of active VEMs attached to VSM

    VMS process retrieves which VSM has active VC connectivity

    SNMP Library gets the last configuration time

    Runs on both primary and secondary VSM

    Heartbeats

    VSM-VSM every second. Drop after 6 missed

    VSM-VEM every second. Drop after 15 missed

    32

    SV2# show system internal redundancy trace

    1 0s START_THREAD ST_NP ST_NP ST_INVALID

    2 0s CP_STATUS_CHG ST_INIT ST_NP ST_INIT

    3 0s SET_VER_RCVD ST_INIT ST_NP ST_INIT

    4 0s STATE_TRANS ST_INIT ST_INIT ST_INIT EV_OS_INIT ST_AC_INIT

    5 0s CP_STATUS_CHG ST_AC ST_INIT ST_AC_INIT

    6 0s STATE_TRANS ST_AC ST_SB ST_AC_INIT EV_OS_SB ST_AC_SB

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Split Brain Recovery for ESXi

    Redundancy Manager in SV2(2.2)

    Module Count

    vCenter Status

    Last Configuration Time

    Last Standby-Active Switch(VSM with longer primary active time)

    Out-of-Sync / Split-Brain causes VSM to reload

    33

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    When does a VEM switch VSMs?

    What if we have two active VSMs?

    What causes a VEM to switch?

    Standby VSM becomes active and broadcasts to all VEMs

    VEM will attach depending on Connectivity between VEM and VSM

    VEM receives the request to switch

    VEM goes into headless mode after 15 seconds

    If a VEM is headless traffic forwarding continues!

    vMotion/Live Migration is blocked

    34

  • Upgrades

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Upgrades

    First always read and follow the upgrade guides

    Go in order

    Take a backup of the VSMs

    On ESXi use the clone to template option

    On Nexus 1x10s use the export function

    Backup the running-config

    Generate a Tech-Support before the upgrade

    If something goes wrong STOP and call TAC

    Use a maintenance window

    VEM upgrades require ESXi hosts to be in Maintenance Mode

    36

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Supported Upgrades

    Starting Version 1.4 1.5 2.1 2.2 Combined

    VMware Upgrade

    Notes

    1.3 Yes 1.4 first* 1.4 first* 1.4 first* No

    1.4 Yes Yes Yes No 1.4 last version

    supporting

    ESX 4.0

    1.5 Yes Yes Yes 1.5.2 for

    combined

    2.1 Yes Yes 2.1 last version

    supporting

    ESX 4.1

    37

    Upgrade matrix: http://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.html

    * Must upgrade to 1.4b first

    http://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.htmlhttp://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.htmlhttp://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.htmlhttp://www.cisco.com/web/techdoc/n1kv/upgrade/utility/n1kvmatrix.html

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Upgrades to 2.2

    Scalability limits may require changes to the VM settings

    For full scalability support:

    CPU reservation to 2GHz

    Memory to 3GB

    VSMs do NOT support multiple vCPUs

    Steps

    Shutdown Secondary VSM

    Make VM changes

    Power Secondary on

    System Switchover

    Repeat steps on Primary VSM

    API can be upgraded individually now

    show plugin status

    38

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Upgrading the VSM

    Changes from 2.1

    VSMs can run newer software than VEMs. New features disabled until VEMs upgraded.

    ISSU upgrade is similar to other Nexus switches

    Copy new kickstart and system images to bootflash

    Run install all command Verifies software compatibility

    Copies images to secondarys bootflash.

    Upgrade/Reboot the Secondary VSM

    Switchover to Secondary VSM Its now the active VSM with VEMs attached

    Upgrade/Reboot the old-Primary VSM

    Requires no outage of the VSM

    Change CPU/Memory after the SV2(2.2) upgrade is complete

    39

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Troubleshooting VSM Upgrades

    If something is wrong after the VSM upgrade STOP

    Call TAC

    Rollback using backup method Shutdown the VSM VMs

    Power-on the Clones (ESXi), Import the backup (Nexus 1x10)

    Changing boot variables to older image is not supported but often works

    Sometimes the VEM wont connect to the Standby VSM

    Try a system switchover once the old primary is upgraded

    Might want to verify Standby VSM before upgrade

    Make sure VEMs can connect to standby

    Use system switchover command

    40

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Upgrading the VEMs

    VEM module upgrade kicked off on VSM

    If VUM is installed everything is automatic VSM communicates with vCenter to manage the upgrade

    Host is placed in maintenance mode(if DRS is installed VMs are migrated off)

    VEM is upgraded and host exits maintenance mode

    Moves on to the next host

    If VUM is not installed Still initiate the process on the VSM

    User manually places ESXi hosts in maintenance mode

    Upgrade the VEM with esxcli command

    Exit maintenance mode and move to the next host

    Always complete the upgrade

    Issue the vmware vem upgrade complete command

    Signals vCenter to use the new VEM VIB when hosts are added

    41

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Troubleshooting VEM Upgrades

    Remember the VMware admin has to acknowledge upgrade in vCenter

    Dont upgrade the VEMs by pushing a baseline

    Make sure you have DRS capacity

    Need to be able to handle one ESXi host in maintenance mode

    If a particular ESXi host fails

    Its usually because the host cannot go into maintenance mode

    From vCenter attempt to put the host in maintenance mode Troubleshoot any issues that prevent it

    If an ESXi host is running a vCenter VM this can cause problems

    You can restart the VEM upgrade after it fails

    It will only upgrade hosts that did not succeed

    42

  • Virtual Ethernet Module Deployment and Troubleshooting

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM Deployment Best Practices

    Again we recommend L3 Control

    L3 control requires a VMKernel NIC on N1K DVS

    We need an L3 interface to forward control traffic

    10/100ms latency for local vs. branch office

    Recommend using the ESXi management VMKernel NIC

    Requires management interface to the VEM

    Doesnt require static routes on ESXi hosts

    Dont create an L3 vmk on same subnet as mgmt vmk

    Dont use UCS Dynamic vNICs in Service-Profiles

    VEM and VM-FEX are mutually exclusive

    44

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM Deployment vEth Port-Profile

    vmk0 interface needs to be migrated to this port-profile

    It must have capability l3control and system VLAN

    Each VMKernel VLAN needs a different port-profile

    VSM only permits VMKs to connect to this port-profile

    45

    port-profile type vethernet vmk-l3

    capability l3control

    vmware port-group

    switchport mode access

    switchport access vlan 119

    capability vxlan

    no shutdown

    system vlan 119

    state enabled

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM Deployment Uplink Port-Profile

    Typically a trunk Verify upstream switch allowed VLAN list matches

    Must have system vlans & a port-channel defined

    MTU must match. Especially important when using OTV.

    46

    port-profile type ethernet system-uplink

    vmware port-group

    switchport mode trunk

    switchport trunk allowed vlan 119,199,219,319

    mtu 9000

    channel-group auto mode on mac-pinning

    no shutdown

    system vlan 119,319

    state enabled

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM L3 Troubleshooting

    1. VMK migrated behind VEM?

    2. VSM-ESXi connectivity?

    Static route needed?

    3. L3 vEth Port-Profile correct?

    4. Uplink Port-Profile correct?

    5. Check the Opaque Data

    6. Check Heartbeats

    47

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM Troubleshooting VSM Connectivity

    VEM adds in vCenter but does not show up on VSM show module

    With L3 its usually an IP routing problem

    If you can ping from VSM to VMK interface then VEM should connect.

    Troubleshoot as you would all VMware L3 issues

    With L2 most of the time its a Control VLAN issue

    Verify Control VLAN connectivity in upstream network

    Check upstream switches for VEM AIPC MAC address

    Additional Information in Appendix 2

    48

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM Deployment VMKs on same subnet

    Dont use multiple VMKs on the same subnet on different virtual switches

    VMware uses a single TCP/IP stack for all VMK interfaces

    No way to pin traffic to an uplink interface.

    One interface gets picked for all traffic on that subnet

    Check out VMware KB article 2010877

    Only one gateway per host

    50

    VMware ESX

    VEM-1

    VMK1

    192.168.10.200

    VMK0

    192.168.10.100

    vSwitch

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Setting Verification

    Verify the VRF

    Can the VSM ping the VEM

    Check SVS domain

    52

    SV2# show ip route vrf management

    0.0.0.0/0, ubest/mbest: 1/0

    *via 14.17.119.254, mgmt0,

    [1/0], 6d20h, static

    SV2# ping 14.17.219.22

    PING 14.17.219.22 (14.17.219.22): 56 data bytes

    64 bytes from 14.17.219.22: icmp_seq=0 ttl=62 time=1.254 ms

    64 bytes from 14.17.219.22: icmp_seq=1 ttl=62 time=1.057 ms

    64 bytes from 14.17.219.22: icmp_seq=2 ttl=62 time=1.055 ms

    SV2# sh svs domain

    SVS domain config:

    Domain id: 1919

    Control vlan: NA

    Packet vlan: NA

    L2/L3 Control mode: L3

    L3 control interface: mgmt0

    Status: Config push to

    VC successful.

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Check Opaque Data

    Opaque data is bootstrap information for the VEM

    Pushed via SCVMM or vCenter during Host Add to DVS

    Is the right Opaque data getting pushed to the ESXi host?

    53

    Should match VLAN defined

    in vEth Port-Profile

    Should match MAC of

    control 0 or mgmt 0

    ~ # vemcmd show card Card UUID type 2: 9aed7c30-84f8-11e2-1234-ff987600005f

    Card name:

    Switch name: SV2

    Switch alias: DvsPortset-0

    Switch uuid: b2 40 3c 50 72 8e 15 f5-6a 3c 7f d1 c6 13 70 cd

    Card domain: 1919

    Card slot: 3

    VEM Tunnel Mode: L3 Mode

    L3 Ctrl Index: 49

    L3 Ctrl VLAN: 119

    VEM Control (AIPC) MAC: 00:02:3d:17:7f:02

    VEM Packet (Inband) MAC: 00:02:3d:27:7f:02

    VEM Control Agent (DPA) MAC: 00:02:3d:47:7f:02

    VEM SPAN MAC: 00:02:3d:37:7f:02

    Primary VSM MAC : 00:02:3d:70:1f:07

    Primary VSM PKT MAC : 00:02:3d:70:1f:08

    Primary VSM MGMT MAC : 00:02:3d:70:1f:06

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    View Heartbeat Messages on VEM

    Use vempkt on the ESXi host vempkt capture [egress|ingress] vlan 119 ltl 50

    Run for 10s to capture several heartbeat cycles

    vempkt cancel capture all

    vempkt display detail all

    vempkt can now export to a pcap file vempkt pcap export

    Look for heartbeat messages on VSM

    54

    SV2# show module vem counters

    --------------------------------------------------------------------------------

    Mod InNR OutMI InMI OutHBeats InHBeats InsCnt RemCnt Crit Tx Errs

    --------------------------------------------------------------------------------

    3 5086 2 2 593401 348535 2 1 0

    4 5 4 4 593401 593296 4 3 0

    5 0 0 0 593401 0 0 0 0

    6 105 4 4 593401 591303 4 3 0

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VEM Troubleshooting - vemlog

    Used for detailed debugging of programming and packet flows

    Executed on the Hypervisor Host

    Enable different debug options to help troubleshoot

    LACP

    QOS

    VXLAN

    IGMP

    VSMVEM Data

    http://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtml

    55

    ~ # vemlog show debug | grep lacp

    Module Available Printing

    sflacp ENWID PL (223) ( 0)

    sf_lacp_pdu_utils ENWID PL (223) ( 0)

    sflacp_hostdata ENWID PL (223) ( 0)

    ~ # vemlog debug sflacp all

    ~ # vemlog show debug | grep lacp

    sflacp ENWID PL (223) ENWIDTPL (255)

    sf_lacp_pdu_utils ENWID PL (223) ( 0)

    sflacp_hostdata ENWID PL (223) ( 0)

    http://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtmlhttp://www.cisco.com/en/US/products/ps9902/products_tech_note09186a0080bed119.shtml

  • Port-Profiles Deploying and Troubleshooting

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port-Profiles

    Usage

    Port-Profile Port-profiles

    vEthernet

    VM vmk l3control

    / vservice

    Ethernet

    UPLINK

    vEthernet PP (default)

    -Virtual Interfaces (vEth x/) (VMs, VMK)

    -Typically Access Ports

    -Configuration: VLAN, ACL, Pinning, QoS

    Ethernet PP

    -Physical Interfaces (Eth x/y)

    -Typically Trunk (could also be access)

    -Configuration: Port-Channel, ACLs, QoS

    57

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Switch Interface Types

    Ethernet Port (eth)

    Correspond to the physical NIC interfaces leaving the server

    Specific to each module or VEM

    VMwares vmnicX == Cisco ethx/y

    Up to 32 physical ports supported per host

    Port Channel (port-channel)

    Aggregation of physical Ethernet ports

    Up to eight Port Channels per host

    Virtual Ethernet Port (vEth)

    One per virtual NIC interface (vNIC) including service console / vmknic

    Notation is VethX

    No module number is assigned to keep naming persistent as VMs move between modules (hosts/VEMs)

    58

    VM1 VM2

    Eth3/1 Eth3/2

    Po1

    Veth2 Veth1

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Loop Prevention without STP

    59

    Cisco VEM

    VM1 VM2 VM3 VM4

    Cisco VEM

    VM5 VM6 VM7 VM7

    Cisco VEM

    VM9 VM10 VM11 VM12

    BPDUs are Dropped

    Eth4/1 Eth4/2

    X

    No Switching from

    Physical NIC

    to NIC

    dj vu check

    Frames with local

    MAC Dropped on

    Ingress

    X

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Spanning-tree and BPDU Best Practice

    Mandatory Spanning-Tree settings per port

    IOS set STP portfast cat65k-1(config-if)# spanning-tree portfast trunk

    NXOS set port type edge n5k-1(config-if)# spanning-tree port type edge trunk

    Highly Recommended Global BPDUFilter/BPDUGuard

    IOS cat65k(config)# spanning-tree portfast bpdufilter

    cat65k(config)# spanning-tree portfast bpduguard

    NXOS n5k-1(config)# spanning-tree port type edge bpduguard default

    n5k-1(config)# spanning-tree port type edge bpdufilter default

    BPDU Filter is mandatory for LACP port-channels

    Set per-port BPDU Guard when Global is not possible

    60

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Ethernet (uplink) Port-Profile Troubleshooting

    Port-Profiles with multiple NICs need a port-channel

    Causes duplicate packets

    Kicks in dj vu driver Requires extra CPU processing

    Fills the logs

    When in doubt, use mac-pinning

    Also same issue if you overlap VLANs in different Port-Profiles on same host

    61

    port-profile type ethernet uplink-nopc

    vmware port-group

    switchport mode trunk

    switchport trunk allowed vlan 1-3967,4048-4093

    no shutdown

    system vlan 11

    state enabled

    port-profile type ethernet uplink-nopc

    vmware port-group

    switchport mode trunk

    switchport trunk allowed vlan 1-3967,4048-4093

    channel-group auto mode on mac-pinning

    no shutdown

    system vlan 11

    state enabled

    WRONG RIGHT

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Nexus 1000V System VLANs

    System VLANs enable interface connectivity before an interface is programmed

    System port-profiles become part of the opaque data

    VEM will load system port-profiles and pass traffic even if VSM is not up

    Unprotected (No ACLs, VSG) before module registers for first time

    Addresses chicken and egg issue

    VEM needs to be programmed, but it needs a working network for this to happen

    Port profiles that contain system VLANs are system port profiles

    Allowed 32 port-profiles with system VLAN

    62

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    System VLAN Guidelines

    The system VLAN must be a subset of the allowed VLAN list on trunk ports

    Only one system VLAN on an access port

    The no system vlan command only when no interface is using the profile

    Once a system profile is in use by at least one interface

    Can add to the list of system VLANs

    Cannot delete VLANs from the list reason to limit usage

    System vlans must be set on egress and ingress port-profiles

    Required System VLANs

    Control, Packet, IP Storage, VMKernel, vCenter, any Management Networks

    63

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VMware DVS Max-Port Issues

    Default to 32 max-ports per port-profile

    Counts toward the maximum number of VMware DVS ports

    8192 by default

    Pre-Provisioned

    Some ports are consumed when you add an ESX host to the DVS

    Two methods to remedy:

    Max-ports under svs connection Allows you to increase the ports of the VMware DVS

    Port-binding auto expand in veth port-profiles N1KV dynamically adds ports as VMs are added

    Set port-binding as default with port-profile default port-binding static auto expand

    64

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Microsoft Network Load Balancing Support

    Unicast mode is officially supported method

    no mac auto-static-learn in vEth port-profile

    Multicast Mode

    NLB virtual cluster address requires a static ARP entry on the edge router

    Works through flooding

    Multicast Mode IGMP

    Disable IGMP snooping on the N1KV

    Upstream switches enable IGMP snooping

    Enable IGMP Querier in the environment

    NLB virtual cluster address requires a static ARP entry on the edge router

    CSCue32210 - Add support for Microsoft NLB - Multicast+IGMP

    65

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Jumbo Frames Support

    System jumbo mtu 9000 Enabled globally by default in SV1(4)+

    Sets the systemwide jumbo MTU size

    Generally do not need to change

    vEthernet ports are 9000 by default

    MTU setting for ethernet type port-profile

    Simply use mtu size in port-profile and nothing else

    Still need to configure upstream network devices

    UCS System QoS Class

    UCS vNIC QoS Policy

    Nexus 5k / 7k / etc

    66

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port-Profile Using Weighted QOS

    Configuration Steps to limit vMotion traffic

    68

    n1kv-l3(config)# class-map type queuing match-all vmotion-class

    n1kv-l3(config-cmap-que)# match protocol ?

    n1k_control N1K control traffic

    n1k_mgmt N1K management traffic

    n1k_packet N1K inband traffic

    vmw_ft VMware fault tolerance traffic

    vmw_iscsi VMware iSCSI traffic

    vmw_mgmt VMware management traffic

    vmw_nfs VMware NFS traffic

    vmw_vmotion VMware vmotion traffic

    n1kv-l3(config-cmap-que)# match protocol vmw_vmotion

    n1kv-l3(config-cmap-que)# policy-map type queuing vmotion-policy

    n1kv-l3(config-pmap-que)# class type queuing vmotion-class

    n1kv-l3(config-pmap-c-que)# bandwidth percent 50

    n1kv-l3(config)# port-profile type eth uplink-vpc

    n1kv-l3(config-port-prof)# service-policy type queuing output vmotion-policy

  • Port Channels

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels

    LACP Port-Channels

    Requires upstream switch support and configuration

    VPC MAC Pinning

    Works with any upstream switch

    Allows for pinning of vEths (VM) to specific links

    VPC Host Mode CDP/Manual (deprecated)

    NIC association is either Manual or CDP

    70

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels

    Best Practice Configuration Guide

    http://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtml

    All Ethernet Port-Profiles must be configured in a Port-Channel

    LACP & MAC-Pinning are recommended modes

    Use Manual/Static Pin Group for granular traffic steering

    Use Manual/Static Pin Groups with multiple vMotion VMKs in ESX 5.x

    Same link-speed for all members. No mixing 1G+10GE+40GE interfaces.

    71

    http://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtmlhttp://www.cisco.com/en/US/products/ps9902/products_configuration_example09186a0080c1ee1e.shtml

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels Best Practice

    If the upstream switch can be clustered (VPC, VBS Stack, VSS) use LACP

    If you are using LACP also use LACP Offload

    UCS-B must use MAC-Pinning

    If the upstream switch can NOT be clustered use MAC-PINNING

    Create channel-groups in port-profile

    Let VSM build the interface port-channel & add physical NICs

    All physical switch ports in port-channel configured identical

    72

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels MAC Pinning

    MAC Pinning provides the dynamism of vPC Host-Mode without requiring CDP to be configured on the upstream switch

    vSphere

    VM VM VM VM

    sys-uplink

    The VM MAC address is used to select link.

    port-profile type ethernet uplink

    vmware port-group

    switchport mode trunk

    switchport trunk allowed vlan 1-10

    channel-group auto mode on mac-pinning

    no shut

    state enable

    system vlan 10

    73

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels MAC Pinning (Link Failure)

    If a failover occurs, all the traffic pinned to an interface will be migrated to the other interfaces. VEM sends GARP to flush upstream CAM tables.

    vSphere

    VM VM VM VM

    sys-uplink

    The VM MAC address is used to select link.

    port-profile type ethernet uplink

    vmware port-group

    switchport mode trunk

    switchport trunk allowed vlan 1-10

    channel-group auto mode on mac-pinning

    no shut

    state enable

    system vlan 10

    74

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels MAC Pinning

    Use Network State Tracking (NST) to detect non-link failures

    Each Eth interface added is a unique Service Group

    SGID # assigned based off vmnic#

    Use pinning id command under vEthernet Port-Profile

    Pins the VM to a particular uplink

    Ordered list for backup

    n1kv(config-port-prof)# pinning id 0 backup 1 2

    Default assignment is Round Robin to an SGID

    New command to make SGID # relative

    n1kv(config-port-prof)# channel-group auto mode on mac-pinning relative

    75

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    MAC Pinning (Host Pinning Tables) n1kv# sh port-channel summary

    1 Po1(SU) Eth NONE Eth5/1(P) Eth5/2(P)

    2 Po2(SU) Eth LACP Eth6/1(P) Eth6/2(P)

    3 Po3(SD) Eth NONE Eth3/3(r)

    [[email protected] ~]# vemcmd show channel type

    LTL Channel_Type

    ------------------

    17 MAC Pinning

    18 MAC Pinning

    76

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    MAC Pinning (Host Pinning Tables) [[email protected] ~]# vemcmd show port

    LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type

    17 Eth3/1 UP UP F/B* 561 0 vmnic0

    18 Eth3/2 UP UP F/B* 561 1 vmnic1

    49 Veth1 UP UP FWD 0 1 vmk0

    [[email protected] ~]# vemcmd show pc

    pce_ind chan pc_ltl pce_in_pc LACP SG_ID NumVethsPinned mbrs

    ------- ---- ------ --------- ---- ----- -------------- ----

    0 1 305 0 N 0 2 17,

    1* 3 18,

    [[email protected] ~]# vemcmd show pinning

    LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name

    10 0 305 32 1 0

    12 0 305 32 1 0

    49 1c0000a0 305 32 1 0 vmk0

    50 1c0000d0 305 32 0 0 vmk1

    77

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port Channels How to Tell Pinning

    Can run from the VSM now

    No need to run command on the VEM

    78

    n1kv-l3# show int virtual pinning module 5

    ------------------------------------------------------

    Veth Pinned Associated PO List of

    Sub Group id interface Eth interface(s)

    ------------------------------------------------------

    Veth2 0 Po5 Eth5/1

    Veth4 2 Po5 Eth5/3

    Veth5 0 Po5 Eth5/1

    Veth6 2 Po5 Eth5/3

    Veth7 0 Po5 Eth5/1

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Static Pinning to Sub-Group

    Static Pinning is similar to VMwares vSwitch active/standby design.

    vmk0 VMotion

    Sub-group 0 Sub-group 2

    Port-channel

    C

    P

    port-profile type ethernet uplink

    channel-group auto mode on mac-

    pinning relative

    port-profile vmkernel

    pinning sub-group id 0 backup 2 1

    port-profile vmkernel

    pinning sub-group id 0 backup 2 1

    port-profile type ethernet vmotion

    pinning sub-group id 2

    vmk0 VMotion

    Sub-group 2

    Port-channel

    C

    P

    After

    failover

    Sub-group 1 Sub-group 1

    79

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    LACP Port Channels

    Use when single upstream or clustered (vPC,VSS, Catalyst Stack) switch

    Use channel-group auto mode active on N1KV

    Use channel-group # mode active/passive on upstream switch

    Switchports must be configured with

    spanning-tree portfast trunk

    spanning-tree bpdufilter enable

    Not compatible with Network State Tracking(NST) with LACP

    80

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port-Channels - LACP

    LACP allows traffic from each VM to fully utilize multiple links simultaneously.

    Allows faster VMotion and faster VM connectivity by using flow based hashing

    port-profile type ethernet uplink

    vmware port-group

    switchport mode trunk

    switchport trunk allowed vlan 1-10

    channel-group auto mode active

    no shut

    state enable

    vSphere

    VM VM VM VM

    Port-channel

    LACP

    Upstream switch clustered (vPC,VSS,VBS,Stack)

    81

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    LACP Troubleshooting

    Do not use Network State Tracking(NST) with LACP

    LACP Port-Channel configured on the upstream switches

    Port-profile created with channel-group auto mode active

    On the VEM

    vemcmd show lacp

    On the VSM and Upstream Switch

    show port-channel summary

    show lacp counters/neighbor Are you seeing LACP PDUs?

    82

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    LACP Debugging ~ # vemcmd show lacp

    LACP Offload is Enabled

    ---------------------------------------------------

    LACP Offload Config for LTL 17

    ---------------------------------------------------

    Channel No : 8

    Channel Mode : Active

    Port Priority : 0x8000

    LACP Bit Set : Yes

    SV2# show lacp counters

    LACPDUs Marker Marker Response LACPDUs

    Port Sent Recv Sent Recv Sent Recv Pkts Err

    ---------------------------------------------------------------------

    port-channel8

    Ethernet10/3 8353 8356 0 0 0 0 0

    Ethernet10/1 8353 8356 0 0 0 0 0

    83

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    LACP Debugging ~ # vemlog show debug | grep lacp

    sflacp ENWID P ( 95) ENW ( 7)

    sf_lacp_pdu_utils ENWID P ( 95) ENW ( 7)

    sflacp_hostdata ENWID P ( 95) ENW ( 7)

    Debug (LTL 16, DIR TX) : Actorstate=7 agg=1 insync=0 coll=0 dis=0 active=1

    short_timeout=1 Port ID (0x8000.0x602), Key (7)

    Debug (LTL 16, DIR TX) :Partnerstate=2 agg=0 insync=0 coll=0 dis=0 active=0

    short_timeout=1 Port ID (0x0.0x0), Key (0)

    Debug sf_lacp_tx_pdu_to_upstream: LTL = 18

    Debug sf_lacp_tx_pdu_to_upstream, NEW LACP PKT : Src(1), Dst(18), VLAN(1),

    FLAGS(1)

    []

    Debug (LTL 18, DIR RX) :Partnerstate=3d agg=1 insync=1 coll=1 dis=1

    active=1 short_timeout=0 Port ID (0x8000.0x602), Key (7)

    Debug (LTL 16, DIR TX) : Actorstate=3d agg=1 insync=1 coll=1 dis=1 active=1

    short_timeout=0 Port ID (0x8000.0x602), Key (7)

    84

  • Virtual Extensible LAN (VXLAN)

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Virtual Extensible Local Area Network (VXLAN)

    Ethernet in IP overlay network

    Entire L2 frame encapsulated in UDP (port 4789)

    50 bytes of overhead

    Include 24-bit VXLAN Identifier

    16 M logical networks

    Mapped into local bridge domains

    Unique multicast group per segment

    VXLAN can cross Layer 3

    Tunnel between VEMs VMs do NOT see VXLAN ID

    Egress to Non-VXLAN network

    87

    Outer

    MAC

    DA

    Outer

    MAC

    SA

    Outer

    802.1Q Outer

    IP DA

    Outer

    IP SA

    Outer

    UDP

    VXLAN ID

    (24 bits)

    Inner

    MAC

    DA

    InnerM

    AC

    SA

    Optional

    Inner

    802.1Q

    Original

    Ethernet

    Payload CRC

    VXLAN Encapsulation Original Ethernet Frame

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Virtual Extensible Local Area Network (VXLAN)

    Each overlay network is known as a VXLAN segment

    Each VXLAN segment identified by a 24-bit segment ID (VNI)

    VXLAN traffic carried between VXLAN Tunnel Endpoints (VTEP)

    VEM module acts as the VTEP

    VM traffic is carried over point to point tunnels between VTEPs

    VM to VM traffic is encapsulated in a VXLAN header

    1550 MTU for encapsulation overhead

    Encapsulated multicast is always flooded No IGMP in VXLAN

    88

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Deployment Modes: Multicast or Unicast?

    Multicast used to be required for unknown broadcast/unicast on VXLAN

    N1KV 2.2 introduced Unicast Mode and Unicast Mac Distribution Mode

    Multicast (VXLAN 1.0)

    Needs Multicast configured throughout complete network

    IGMP Querier in VLAN

    Multicast routing and proxy ARP across subnets

    VTEPs all join multicast group

    Interoperates with N9K, CSR1K, other Nexus products

    Unicast Mode (VXLAN 1.5)

    VEMs flood each other directly for unknown broadcast/unicast

    Keep a list of other VEMs in each VXLAN

    89

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Deployment Modes: When to use MAC Distribution?

    MAC distribution will provide best performance

    No Flooding & Learning

    Full MAC table distributed to each VEM

    VEMs report local MACs to VSM

    VSM distributes {MAC,VTEP} mapping to each VEM

    VXLAN traffic cannot span multiple Nexus 1000V switches*

    Two caveats

    No vEth VXLAN trunk mode support with MAC distribution

    Wont work with Microsoft NLB

    90

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Forwarding Basics

    VEM 1 VEM 2

    Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn

    VEM learns VMs Source (MAC, Host VXLAN IP) tuple

    Broadcast, Multicast, and Unknown Unicast Traffic

    VM broadcast & unknown unicast traffic are sent as multicast

    Unicast Traffic

    Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM)

    VM VM VM VM

    92

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Enhanced VXLAN

    VXLAN

    (multicast mode)

    Enhanced VXLAN

    (unicast mode)

    Enhanced VXLAN

    MAC Distribution

    Enhanced VXLAN

    ARP Termination

    Broadcast /

    Multicast

    Multicast

    Encapsulation

    Replication plus

    Unicast Encap

    Replication plus

    Unicast Encap

    Replication plus

    Unicast Encap

    Unknown Unicast

    Multicast

    Encapsulation

    Replication plus

    Unicast Encap

    Drop Drop

    Known Unicast Unicast

    Encapsulation Unicast Encap Unicast Encap Unicast Encap

    ARP Unicast

    Encapsulation

    Replication plus

    Unicast Encap

    Replication plus

    Unicast Encap VEM ARP Reply

    96

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Configuration: Unicast

    VMkernel interface acts as VTEP

    VSM Control Mode should be L3

    Bridge domain is configured as Unicast or Unicast Mac Distribution

    97

    feature segmentation

    feature vxlan-gateway

    port-profile type vethernet vmk-l3-vxlan-vtep

    capability l3control

    vmware port-group

    switchport mode access

    switchport access vlan 119

    capability vxlan

    no shutdown

    system vlan 119

    state enabled

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Bridge Domain Configuration: Unicast

    Create a bridge-domain in unicast mode

    Scenario 1:

    Scenario 2:

    switch(config)# segment mode unicast-only (Global)

    switch(config)# bridge-domain segment-cisco

    switch(config-bd)# segment id 5000

    switch(config-bd)# segment distribution mac

    switch(config)# bridge-domain segment-cisco

    switch(config-bd)# segment id 5000

    switch(config-bd)# segment mode unicast-only (Per BD override)

    switch(config-bd)# segment distribution mac

    98

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Port-Profile Configuration

    Create an Access Port-Profile with the VXLAN Bridge Domain

    Assign to VMs in vCenter port-profile type vethernet bd-5000

    vmware port-group

    switchport mode access

    switchport access bridge-domain bd-5000

    no shutdown

    state enabled

    99

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Debugging SV2# show bridge-domain bd-5000

    Bridge-domain bd-5000 (2 ports in all)

    Segment ID: 5000 (Manual/Active)

    Mode: Unicast-only (override)

    MAC Distribution: Disable (override)

    Group IP: NULL

    State: UP Mac learning: Enabled

    Veth9, Veth45

    SV2# show bridge-domain bd-5000 vteps

    Bridge-domain: bd-5000

    VTEP Table Version: 21

    Port Module VTEP-IP Address VTEP-Flags

    ---------------------------------------------------------------------------

    Veth1 3 14.17.119.21 (D)

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Debugging ~ # vemcmd show vxlan-vteps

    Bridge-Domain: bd-5000 Segment ID: 5000

    Designated Remote VTEP IPs (*=forwarding publish incapable):

    14.17.119.22(DSN: 1), 14.17.119.36(DSN: 1)*

    ~ # vemcmd show bd bd-name bd-5000

    BD 31, vdc 1, segment id 5000, segment group IP 0.0.0.0,

    encap VXLAN, vff_mode Anycast,swbd 4096, VLAN 0, 1 ports, "bd-5000"

    Segment Mode: Unicast

    VTEP DSN: 1 , MAC DSN: 0

    Portlist:

    52 win2k.eth0

    Virtual Machine

    in VXLAN 5000

    101

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Debugging ~ # vemcmd show l2 segment 5000

    Bridge domain 31 brtmax 4096, brtcnt 3, timeout 300

    Segment ID 5000, swbd 4096, "bd-5000"

    Flags: P - PVLAN S - Secure D - Drop

    Type MAC Address LTL timeout PVLAN Remote IP DSN

    Dynamic 00:50:56:bc:73:1a 561 121 14.17.119.22 0

    Static 00:50:56:a9:00:2e 52 0 0.0.0.0 0

    Dynamic 54:7f:ee:2f:33:81 561 2 14.17.119.36 0

    ESXi Host #2

    VXLAN

    Gateway

    102

  • Nexus 1010 and 1110

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSM Deployment Scenarios Nexus 1110

    VSM on a Nexus 1010/X or 1110-S/X

    Its still a Virtual Machine

    Up to 14 VSM pairs on one 1110-X cluster

    Always deploy in the appliance pairs!

    N110 allows for Network team to own the virtualization platform

    N110s should go in the Aggregation Layer

    Stretched Model requires

    L2 Connectivity

    10ms latency

    Cisco Cloud Services Platform

    104 *Next Release

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    1110-S/X Deployment Scenario

    105

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Cloud Services Platform (CSP)

    Based off UCS C2x0 M3 server

    Same CIMC/BIOS/firmware

    Provide 6 x 1G network connections

    1110-X 2 x 10G - SP1(7) 10G available only on purchase. No upgrade available.

    Encryption Accelerator Card for Citrix VPX SP1(7)

    Virtual Service Blade (VSB) Support

    1010/1110-S supports up to 10

    1010/1110-X supports up to 14

    Nexus 1010/1010-X/1110-S/1110-X

    106

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Cloud Services Platform (CSP)

    Current supported VSBs

    Nexus 1000V VSM (ESX/HyperV/KVM)

    Virtual Security Gateway (VSG)

    Network Analysis Module (NAM)

    Data Center Network Manager (DCNM)

    Citrix NetScaler VPX

    107

    VSB Minimum Version

    HyperV SP(6.1)

    VXLAN GW SP1(6.1)

    Citrix Netscaler SP1(6.2)

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Cisco Cloud Services Platform (CSP)

    Must be deployed in pairs

    No option for standalone

    Deploy in the Aggregation Layer

    Must be in the same L2 domain for management and control

    Can be geographically diverse

    Uses same HA mechanism as VSM with domain-id and control vlan

    Do not overlap the domain-id between a 1x10 and a VSM

    Whats not supported?

    Primary and Secondary VSM on same 1x10

    Primary VSM on ESX and Secondary VSM on 1x10 or vice versa

    108

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VSB Backups using Import/Export

    Works with VSM, NAM, and VSG

    Can Import/Export both primary and secondary

    Export requires that VSB be shutdown

    Images are stored in export-import/ dir on bootflash

    Can be manually copied off to remote storage

    109

    n1010-1# copy scp://[email protected]/root/Vdisk4.img.tar.00

    bootflash:export-import vrf management

    n1010-1(config)# virtual-service-blade training

    n1010-1(config-vsb-config)# import primary Vdisk4.img.tar.00

    Note: import started..

    Note: please be patient..

    Note: Import cli returns check VSB status for completion

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Network Classes and Topologies

    Management

    Carries the mgmt0 interface of the 1x10

    Carries the mgmt0 traffic for all VSMs installed

    Control

    Carries all the control and packet traffic for the VSMs installed on the 1x10

    Carries control traffic for HA between primary and secondary 1x10

    Data

    Used by Virtual Service Blades other than VSM

    Passthrough

    Binds physical NIC to VSB

    5 Network Topologies choices

    110

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Network Topologies

    Uplink Type Management VLAN Control VLAN Data VLAN

    1 Ports 1 and 2 Ports 1 and 2 Ports 1 and 2

    2 Ports 1 and 2 Ports 1 and 2 (HA) Ports 3-6 (LACP)

    3 Ports 1 and 2 Ports 3-6 (LACP) Ports 3-6 (LACP)

    4 Ports 1 and 2 Ports 3 and 4 Ports 5 and 6

    Flexible There is no traffic segregation based on traffic class.

    *Must use for VXGW deployements.

    111

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Recommendations

    If you are not planning on using other VSBs

    Topology 3 gives best bandwidth and redundancy for control VLAN Negative is that is harder to configure

    If using VXGW, Netscaler, or shared between production / lab network

    Topology 5 is Flexible

    Flexible allows any configuration

    Recommend port-channels

    Remember VSM latency is key over bandwidth

    Use VPC or VSS upstream if you have it

    112

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Participate in the My Favorite Speaker Contest

    Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

    Send a tweet and include

    Your favorite speakers Twitter handle @juicyUCS

    Two hashtags: #CLUS #MyFavoriteSpeaker

    You can submit an entry for more than one of your favorite speakers

    Dont forget to follow @CiscoLive and @CiscoPress

    View the official rules at http://bit.ly/CLUSwin

    Promote Your Favorite Speaker and You Could be a Winner

    113

    http://bit.ly/CLUSwin

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Complete Your Online Session Evaluation

    Give us your feedback and you could win fabulous prizes. Winners announced daily.

    Complete your session evaluation through the Cisco Live mobile app or visit one of the interactive kiosks located throughout the convention center.

    Dont forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

    114

    https://www.ciscolive.com/online

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Continue Your Education

    Demos in the Cisco Campus

    Walk-in Self-Paced Labs

    Table Topics Moscone Center West 3rd Floor Lobby

    Discuss Experiences with Cisco Services with Distinguished Service Engineers

    Meet the Engineer 1:1 meetings

    115

  • Appendix A L2 Troubleshooting

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    L2 Control VEM VSM Troubleshooting Steps

    1. VSM MAC address

    2. VSM is connected to vCenter

    3. VSM has Control VLAN on right interface

    4. Uplink port-profile has Control vlan

    5. VEM sees control VLAN

    6. VEM and VSM see each others MAC

    7. Physical network sees VEM and VSM MAC

    8. VSM sees heartbeat messages from VEM

    119

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    n1kv-l2# show svs neighbors

    Active Domain ID: 422

    AIPC Interface MAC: 0050-56a9-2535

    Inband Interface MAC: 0050-56a9-2537

    Step 1: VSM MAC

    Need for L2 troubleshooting

    On VSM run show svs neighbors

    Its the AIPC Interface MAC

    120

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    n1kv-l2# show svs connections

    connection VC:

    ip address: 172.18.217.241

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: Harrington

    admin:

    max-ports: 8192

    DVS uuid: 3e 80 29 50 ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e

    config status: Enabled

    operational status: Connected

    Step 2: VSM vCenter Connectivity

    Verify VSM is connected to vCenter

    121

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Step 3: Verify VSM VM Control interface

    1st interface listed is Control Interface

    Interface connected?

    122

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Step 4: Verify Uplink Port-Profile

    The first ESX interface added to the N1KV must have Control VLAN

    Verify uplink port-profile has Control VLAN defined and system VLAN

    123

    n1kv-l2# show run port-profile uplink version 4.2(1)SV1(5.1) port-profile type ethernet uplink vmware port-group switchport mode trunk switchport trunk allowed vlan 1-3967,4048-4093 no shutdown system vlan 2 state enabled

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Step 5: Verify VEM Sees Control VLAN

    Verify VEM sees control VLAN with commands

    vemcmd show card

    vemcmd show port

    vemcmd show trunk

    124

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    [~ # vemcmd show card

    Card UUID type 2: 33393138-3335-5553-4537-31314e343636

    Card name: cae-esx-154

    Switch name: n1kv-l2

    Switch alias: DvsPortset-0

    Switch uuid: 3e 80 29 50 ad 9f f9 7f-43 d6 9b 6d a2 af cb 3e

    Card domain: 422

    Card slot: 5

    VEM Tunnel Mode: L2 Mode

    VEM Control (AIPC) MAC: 00:02:3d:11:a6:04

    VEM Packet (Inband) MAC: 00:02:3d:21:a6:04

    VEM Control Agent (DPA) MAC: 00:02:3d:41:a6:04

    ..

    ..

    Card control VLAN: 2

    Card packet VLAN: 2

    Vemcmd show card

    Control, packet vlans and domain-ID match with VSM

    125

    MAC the VSM

    should learn for

    VEM

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    ~ # vemcmd show port-old

    LTL IfIndex Vlan/ Bndl SG_ID Pinned_SGID Type Admin State CBL Mode Name

    SegId

    6 0 1 T 0 32 32 VIRT UP UP 1 Trunk vns

    8 0 3969 0 32 32 VIRT UP UP 1 Access

    9 0 3969 0 32 32 VIRT UP UP 1 Access

    10 0 2 0 32 32 VIRT UP UP 1 Access

    11 0 3968 0 32 32 VIRT UP UP 1 Access

    12 0 2 0 32 32 VIRT UP UP 1 Access

    13 0 1 0 32 32 VIRT UP UP 0 Access

    14 0 3971 0 32 32 VIRT UP UP 1 Access

    15 0 3971 0 32 32 VIRT UP UP 1 Access

    16 0 1 T 0 32 32 VIRT UP UP 1 Trunk ar

    17 25010000 1 T 0 32 32 PHYS UP UP 1 Trunk vmnic0

    Vemcmd show port-old

    Ports with LTLs 8, 9,10 are UP and CBL states are 1.

    ESX Physical ports are UP and CBL states 1.

    126

    Local Target Logic (LTL) is an index to address a port, or group of ports. Data path lookup engine takes LTL

    as input, and gives LTL as output.

    LTL scheme: [0-14: internal ports] [15-271: pNICs,VMs, etc]

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    ~ # vemcmd show trunk

    Trunk port 6 native_vlan 1 CBL 1

    vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1,

    vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1,

    vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,

    Trunk port 16 native_vlan 1 CBL 1

    vlan(1) cbl 1, vlan(3970) cbl 1, vlan(3969) cbl 1, vlan(3968) cbl 1, vlan(3971) cbl 1,

    vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1, vlan(151) cbl 1,

    vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,

    Trunk port 17 native_vlan 1 CBL 1

    vlan(1) cbl 1, vlan(11) cbl 1, vlan(10) cbl 1, vlan(150) cbl 1, vlan(2) cbl 1,

    vlan(151) cbl 1, vlan(152) cbl 1, vlan(153) cbl 1, vlan(154) cbl 1, vlan(155) cbl 1,

    Vemcmd show trunk

    Control and packet are CBL states 1 on the physical ports.

    127

    ~ # vemcmd show port vlans

    Native VLAN Allowed

    LTL VSM Port Mode VLAN State Vlans

    17 Eth5/1 T 1 FWD 2,10-11,150-155

    ~ #

    vemcmd show port vlans

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    ~ # vemcmd show l2 2

    Bridge domain 9 brtmax 4096, brtcnt 32, timeout 300

    VLAN 2, swbd 2, ""

    Flags: P - PVLAN S - Secure D - Drop

    Type MAC Address LTL timeout Flags PVLAN

    Static 00:02:3d:21:a6:04 12 0

    Dynamic 00:50:56:a9:25:35 17 1

    Step 6: VEM and VSM See Each Others MAC

    Is the VEM learning the MAC of the VSM?

    On VEM vemcmd show l2 do you see the mac of the VSM?

    128

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    n1kv-l2# show mac address-table vlan 2

    VLAN MAC Address Type Age Port Mod

    ---------+-----------------+-------+---------+------------------------------+---

    2 0002.3d21.a604 static 0 N1KV Internal Port 5

    2 0002.3d41.a604 static 0 N1KV Internal Port 5

    VEM and VSM See Each Others MAC

    Is the VSM learning the MAC of the VEM?

    129

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    cae-cat6k-1#show mac-address-table vlan 2

    Legend: * - primary entry

    age - seconds since last seen

    n/a - not available

    vlan mac address type learn age ports

    ------+----------------+--------+-----+----------+--------------------------

    * 2 0050.5677.7770 dynamic Yes 360 Gi3/48

    * 2 0050.56a9.2535 dynamic Yes 0 Gi4/9

    * 2 3333.0000.0016 static Yes - Switch,Stby-Switch

    * 2 0002.3d41.a604 dynamic Yes 0 Gi1/4

    Step 7: Physical Switch Mac Table

    Check the physical switch MAC address table

    Are the MACs of the VEM and VSM getting learned by the physical switches in the right VLANs?

    130

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Step 8: VEM VSM Heartbeat

    One Heartbeat per second per VEM from VSM

    Timeout for VEM from VSM is 6 seconds of missed heartbeats

    After 6 seconds VSM will drop VEM

    Use vempkt capture to view heartbeats

    SPAN physical switch ports for heartbeats

    131

  • Appendix B Miscellaneous Commands

  • Appendix C VXLAN Multicast

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Configuration: Multicast

    VMkernel interface to act as VTEP

    VSM Control Mode should be L3

    Multicast for Broadcast traffic

    IP Multicast forwarding is required

    Multicast addresses

    Multiple segments can be mapped to a single multicast group

    If VXLAN transport is contained to a single VLAN, IGMP Querier must be enabled on that VLAN

    If VXLAN transport is traversing routers Multicast routing must be enabled.

    Proxy ARP must also be enabled

    1550 MTU for VXLAN encapsulation overhead

    134

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Configuration: Multicast

    Upstream Switch Configuration

    Enable IGMP Querier

    Set physical switch port MTU to 1550

    Enable proxy-arp on upstream SVI

    ESXi Host

    Create VMK interface for VXLAN

    Nexus 1000V

    Enable feature segmentation

    Create a Bridge Domain

    Create a port-profile for VTEP VMK interface

    Create a veth port-profile for the VMs

    135

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Configuration: Multicast

    Increase the MTU on your eth port-profile

    n1kv-l3(config)# port-profile type eth uplink

    n1kv-l3(config-port-prof)# mtu 1550

    Create veth port-profile for VXLAN VMK interface

    n1kv-l3(config)# port-profile type vethernet VXLAN-VMK

    n1kv-l3(config-port-prof)# switchport mode access

    n1kv-l3(config-port-prof)# switchport access vlan 11

    n1kv-l3(config-port-prof)# no shutdown

    n1kv-l3(config-port-prof)# system vlan 11

    n1kv-l3(config-port-prof)# vmware port-group

    n1kv-l3(config-port-prof)# capability vxlan

    n1kv-l3(config-port-prof)# state enabled

    136

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Configuration: Multicast

    Configure the Bridge Domain

    Maps a segment ID to a multicast address

    Segment ID >4096

    n1kv-l3(config)# bridge-domain vxlan-1

    n1kv-l3(config-bd)# segment id 5000

    n1kv-l3(config-bd)# group 224.3.5.2

    Create VM port-profile

    n1kv-l3(config)# port-profile type veth vm-vxlan-1

    n1kv-l3(config-port-prof)# vmware port-group

    n1kv-l3(config-port-prof)# switchport mode access

    n1kv-l3(config-port-prof)# switchport access bridge-domain vxlan-1

    n1kv-l3(config-port-prof)# no shut

    n1kv-l3(config-port-prof)# state enabled

    137

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Troubleshooting Tips

    Verify your Bridge Domains, VM port-profiles, and VXLAN VMK port-profiles

    Verify multicast on your upstream switches

    show ip igmp snooping

    Do you see the VTEPs

    Use vmkping on the ESXi host to verify network and MTU

    Use 1542 to cover the addition of the ICMP header

    ~ # vmkping -s 1542 -d 1.1.1.1

    Verify the VEM has the right VXLAN capability

    ~ # vemcmd show vxlan interfaces

    LTL IP

    ---------------

    69 1.1.1.2

    138

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Troubleshooting Tips

    ~ # vemcmd show port vlans

    LTL VSM Port Mode VLAN/ State Vlans/SegID

    17 Eth4/1 T 1 FWD 25,626-640

    18 Eth4/2 T 1 FWD 25,626-640

    53 Veth19 A 6000 FWD 6000

    Verify the VEM was programmed correctly

    ~ # vemcmd show segment 6000

    BD 23, vdc 1, segment id 6000, segment group IP 225.6.26.10, swbd 4096, 2 ports, "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18"

    Portlist:

    53 vse-vCDNI-6-26-vl634-backed (b6

    68 vCDNI-2 (5ac7d73c-d1d1-4877-8ef

    139

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Other Useful Commands

    vemcmd show port

    vemcmd show igmp

    vemcmd show l2 segment

    vemcmd show vxlan-encap [ltl/mac]

    vemcmd show vlxan-stats all

    Detailed slides in the Appendix

    140

  • Appendix D - Additional VXLAN TShoot

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Other Useful Commands

    Verify Multicast Upstream Nexus 7K/5K

    Verify querier is enabled for vlan VMK interfaces are on switch# show run

    vlan configuration 634

    ip igmp snooping querier 1.1.1.161

    142

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN other Useful Commands

    Verify IGMP snooping is configured

    CWD.35.04-7000-1# show ip igmp snooping vlan 634

    IGMP Snooping information for vlan 634

    IGMP snooping enabled

    Optimised Multicast Flood (OMF) enabled

    IGMP querier present, address: 1.1.1.161, version: 3, i/f Po1

    Querier interval: 125 secs

    Querier last member query interval: 1 secs

    Querier robustness: 2

    Switch-querier enabled, address 1.1.1.161, currently running

    ..

    IGMPv3 Report suppression disabled

    Link Local Groups suppression disabled

    Router port detection using PIM Hellos, IGMP Queries

    Number of router-ports: 1

    Number of groups: 1

    VLAN vPC function enabled

    Active ports:

    Po1 Po9 Po17 Po25

    Po31 Po52 Po100 Eth2/30

    143

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLN other Useful Commands

    Verify multicast IP address for the VXLAN is being learned

    CWD.35.04-7000-1# show ip igmp snooping groups vlan 634

    Type: S - Static, D - Dynamic, R - Router port, F - Fabricpath core

    port

    Vlan Group Address Ver Type Port list

    634 */* - R Po1

    634 225.6.26.10 v2 D Po100 Eth2/30

    144

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Other Useful Commands

    vemcmd show port

    Will show ports that are on a vxlan ~ # vemcmd show port

    LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type

    17 Eth4/1 UP UP F/B* 305 0 vmnic0

    18 Eth4/2 UP UP F/B* 305 1 vmnic1

    49 Veth2 UP UP FWD 0 0 vmk0

    ...

    53 Veth19 UP UP FWD 0 vse-vCDNI-6-26-vl634-backed (b6

    54 Veth16 UP UP FWD 0 0 vse-vCDNI-6-26-vl634-backed (b6

    ...

    68 Veth21 UP UP FWD 0 vCDNI-2 (5ac7d73c-d1d1-4877-8ef

    69 Veth22 UP UP FWD 0 0 vmk1 vxlan

    305 Po2 UP UP F/B* 0

    145

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Other Useful Commands

    vemcmd show igmp

    Verify that multicast is enabled ~ # vemcmd show igmp 634

    IGMP is ENABLED on VLAN 634

    Multicast Group Table:

    Group */*, Multicast LTL: 4410

    vemcmd show l2 segment

    Verify the VEM is learning MAC addresses in the VXLAN ~ # vemcmd show l2 segment 6000

    Bridge domain 23 brtmax 4096, brtcnt 3, timeout 300

    Segment ID 6000, swbd 4096, "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18"

    Flags: P - PVLAN S - Secure D - Drop

    Type MAC Address LTL timeout Flags PVLAN Remote IP

    Static 00:50:56:01:02:0a 68 0 0.0.0.0

    Dynamic 00:50:56:01:02:09 305 1 1.1.1.1

    Static 00:50:56:01:02:15 53 0 0.0.0.0

    146

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Other Useful Commands

    vemcmd show vxlan-encap [ltl/mac]

    Identify the traffic path a MAC or LTL will utilize ~ # vemcmd show vxlan-encap ltl 68

    Encapsulation details for LTL 68 in BD "dvs.VCDVSvCDNI-6-26-vl634-backed-b69c1d1d-02bf-4581-9b7e-fa06c64e8c18":

    Source MAC: 00:50:56:01:02:0a

    Segment ID: 6000

    Multicast Group IP: 225.6.26.10

    Encapsulating L2 LISP Interface LTL: 69

    Encapsulating Source IP: 1.1.1.2

    Encapsulating Source MAC: 00:50:56:7e:0e:b6

    Pinning of L2 LISP Interface to the Uplink:

    LTL IfIndex PC_LTL VSM_SGID Eff_SGID iSCSI_LTL* Name

    69 1c000150 305 32 0 0 vmk1

    147

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Other Useful Commands

    vemcmd show vlxan-stats all

    Show VXLAN traffic stats ~ # vemcmd show vxlan-stats all

    LTL Ucast Mcast Ucast Mcast Total

    Encaps Encaps Decaps Decaps Drops

    53 67 300 47 0 23

    68 11701 47135 11690 61 12

    69 11768 125793 11737 61 0

    148

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    VXLAN Load Balancing

    With LACP port-channel 5-tuple hash is used

    Use singe VMK VXLAN interface

    VEM does the hashing across all the links

    Remember to change load balancing to 5-tuple hashing On the upstream switch and on the VSM

    With VPC MAC Pinning

    Create a VMK VXLAN interface for each available uplink

    VEM will pin an interface to each available link

    The VEM will distribute the VM's flows between the vmknics based on a hash of the source MAC.

    149

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Verification: Unicast

    Verify the bridge-domain configuration on VSM switch# sho bridge-domain

    Global Configuration:

    Mode: Unicast-only

    MAC Distribution: Disable

    Bridge-domain segment-cisco (3 ports in all)

    Segment ID: 9001 (Manual/Active)

    Mode: Unicast-only (default)

    MAC Distribution: Disable (default)

    Group IP: NULL

    State: UP Mac learning: Enabled

    Veth2, Veth3, Veth5

    Verify the bridge-domain configuration on VEM

    switch# module vem 4 execute vemcmd show bd bd-name segment-cisco

    BD 26, vdc 1, segment id 9001, segment group IP 0.0.0.0, swbd 4102, 2 ports, "segment-cisco"

    Segment Mode: Unicast

    VTEP DSN: 1 , MAC DSN: 1

    Portlist:

    53 RedHat_VM1_112.eth4

    54 RedHat_VM1_112.eth5

    ~ #

    If MAC Distribution is enabled this will be Enable

    If MAC Distribution is enabled this will be

    Segment Mode: Unicast, Mac-Distribution

    VTEP and MAC download sequence numbers should

    be checked against VTEP entries (vemcmd show

    vxlan-vteps) and MAC entries (vemcmd show l2 bd

    bd-name ) respectively

    150

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Verification (Port configuration)

    Verify the port configuration on VSM switch# sho int switchport | begin Vethernet2

    Name: Vethernet2

    Switchport: Enabled

    Switchport Monitor: Not enabled

    Operational Mode: access

    Access Mode VLAN: 0 (none)

    Access BD name: segment-cisco

    [SNIP]

    151

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Verification (Port configuration)

    switch# module vem 4 execute vemcmd show port

    LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type

    17 Eth4/1 UP UP F/B* 561 0 vmnic0

    49 DOWN UP BLK 0 RedHat_VM1_112 ethernet7

    50 Veth8 DOWN UP BLK 0 RedHat_VM1_112.eth8

    51 Veth4 UP UP FWD 0 0 vmk1 VXLAN

    52 DOWN UP BLK 0 RedHat_VM1_112.eth6

    53 Veth2 UP UP FWD 0 RedHat_VM1_112.eth4

    54 Veth3 UP UP FWD 0 RedHat_VM1_112.eth5

    561 Po2 UP UP F/B* 0

    * F/B: Port is BLOCKED on some of the vlans.

    One or more vlans are either not created or

    not in the list of allowed vlans for this port.

    Please run "vemcmd show port vlans" to see the details.

    ~ #

    Verify the port configuration on VEM

    152

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Verification (VTEP Distribution)

    Verify the VTEP distribution on VEM

    switch# sho bridge-domain segment-cisco vteps

    D: Designated VTEP I:Forwarding Publish Incapable VTEP

    Bridge-domain: segment-cisco

    VTEP Table Version: 2

    Ifindex Module VTEP-IP Address

    -----------------------------------------------------------------

    -------------

    Veth4 4 10.106.199.116(D)

    Veth1 5 10.106.199.117(D)

    switch#

    Verify the VTEP distribution on VSM

    switch# module vem 4 execute vemcmd show vxlan-vteps

    Bridge-Domain: segment-cisco Segment ID: 9001

    Designated Remote VTEP IPs (*=forwarding publish incapable):

    10.106.199.117(DSN: 1),

    To be compared with

    echo show vxlan

    version-table output on

    VEM

    Compare against vemcmd show bd bd-name

    VTEP DSN output

    153

  • 2014 Cisco and/or its affiliates. All rights reserved. BRKVIR-3013 Cisco Public

    Verification (MAC table in unicast only mode)

    switch# module vem 4 execute vemcmd show l2 bd-name segment-cisco

    Bridge domain 26 brtmax 4096, brtcnt 3, timeout 300

    Segment ID 9001, swbd 4102, "segment-cisco"

    Flags: P - PVLAN S - Secure D - Drop

    Type MAC Address LTL timeout Flags PVLAN Remote IP DSN

    Dynamic 00:50:56:83:01:4e 561 1 10.106.199.117 0

    Static 00:50:56:83:01:61 54 0 0.0.0.0 0

    Static 00:50:56:83:01:60 53 0 0.0.0.0 0

    switch#

    MAC address table will display remote IP learning in the segment-cisco bridge domain

    154