VXLAN - Bringing Hypervisord2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKDCN-2200.pdf · VXLAN - Bringing Hypervisor & Nexus Together Wayne Davis –Technical Solutions Architect

VXLAN - Bringing Hypervisor & Nexus Together

Wayne Davis – Technical Solutions Architect

BRKDCN-2200

• Ready, Set, Tunnel - VXLAN Refresher

• Design Details - Under the Hood

• Avoid Resume Generating Events

• Best Practices

• Case Study Deployment Scenario's

• Roadmap – Whats Next?

• Wrap It Up

Agenda

BRKDCN-2200

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public

Session Rat hole’s

• Concentrate on deployment VXLAN – F & L

• How to Jumpstart ESX

• Introduction to Nexus 1000v

• All components of the design(s) choices

• Configuration Installation “Gotha's”

• Deep Dive into ACI

• Security “Line by Line cfg”

• Troubleshooting Deep dive Design

4BRKDCN-2200

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 5

Which Encapsulation?

BRKDCN-2200

VXLAN NVGRE

MPLS

FabricPath

LISP


Questions

• Is it a standards based protocol used for traffic flows?

• Would you consider using BGP as a Control Plane in your Data Center ?

• Barriers to Adoption – Configuration Complexity ? Automation help ?

• Importance of being Standards-Based ? Proof of Interoperability

• Reliability and Scale out design *Important*?

• Active/Active Data Center design, is it possible?

6BRKDCN-2200

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 7BRKDCN-2200

Introduction – What is VXLAN ?MAC-in-IP Encapsulation

Outer

MAC

DA

Outer

MAC

SA

Outer

802.1Q

Outer

IP DA

Outer

IP SA

Outer

UDP

VXLAN ID

(24 bits)

Inner

MAC

DA

Inner

MAC

SA

Optional

Inner

802.1Q

Original

Ethernet

Payload

CRC

VXLAN Encapsulation Original Ethernet Frame

CRC

DA

TA

PLA

NE

16 M Segments

A

B

C

Switch

Encap

A

B

C

Switch

DcapIP Network

Ethernet Frames Ethernet Frames

IP/UDP Packets

IP Addr

1.1.1.1

IP Addr

2.2.2.2

NE

TW

OR

K

Tunnel

Endpoints


VXLAN: Flood-&-Learn vs EVPN Control PlaneFlood-&-Learn EVPN Control Plane

Overlay Services L2+L3 L2+L3

Underlay Network IP network with ECMP IP network with ECMP

Encapsulation MAC in UDP MAC in UDP

Peer Discovery Data-driven flood-&-learn MP-BGP

Peer Authentication Not available MP-BGP

Host Route Learning Local hosts: Data-driven flood-&-learn

Remote hosts: Data-driven flood-&-learn

Local Host: Data-driven

Remote host: MP-BGP

Host Route Distribution No route distribution. MP-BGP

L2/L3 Unicast Forwarding Unicast encap Unicast encap

BUM Traffic forwarding Multicast replication

Unicast/Ingress replication

Multicast replication

Unicast/Ingress replication


VXLAN - VTEP

VXLAN terminates its tunnels on VTEPs (VXLAN Tunnel End Point).

Each VTEP has two interfaces, one is to provide bridging function for local hosts, the

other has an IP identification in the core network for VXLAN encapsulation / de-

encapsulate.

Local LAN Segment

IP Interface

End SystemEnd System

VTEP

Transport IP Network

Local LAN Segment

IP Interface

End SystemEnd System

VTEP


VXLAN - BGP-EVPN

Tunnel Endpoints LocationHost Reachability Information

• Mac Address + IP Address

VTEP VTEP VTEP VTEP VTEP

R/R R/RIBGP Route Reflector*

(on spine or different box)

VXLAN OverlayBGP Peers

on VTEPs

Use Multi-Protocol BGP with EVPN Address family for :


VXLAN EVPN - Solution Advantages

Early ARP

Termination

Distributed Anycast

Gateway

Suppresses flooding for Unknown Unicast

ARP

Authenticate Tunnel Endpoints

Seamless and Optimal vm-mobility

Forwarding in the overlay

Active/Active

Multipathing

Active/Active and Resilient Multipathing

using vPC on Nexus

Ingress Replication Unicast Alternative to Multicast underlay

Security

Design Details – Under the Hood


Under the Hood – “Choice”

• N1K -Segmentation

• N9K - EVPN • Application Centric Infrastructure

• VxLAN 1.0 / 2.0

• Multiple OS support

• VSG VM and Custom Attributes

• Appliance based option

• Multi-technology Design

• Requires 9k switches

• Can be upgraded (NxOS to ACI)

• VxLAN GW (anycast)

• Jump Data Centers with L2 domains

• Broadcast suppression

• Supports Any Hypervisor

• Stateful Firewall Support

• Single Pane of Glass Mgmt.

• Container design Model

• Security per vNIC


Under The Hood – Physical

N2K N2K

Chassis

Servers

vPC

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Cisco Nexus 2148T

1GE Fabric Extender

STAT

ID 45 46 47 48 1 2 3 4

Bay1

Bay9

DSModule

PS1

Bay8

Bay16

PS6

HPBladeSystem

c7000Enclosure

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108

OK FAIL OK FAIL OK FAIL OK FAIL

1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16N55-M16UP

1/10 GIGABIT ETHERNET 1/2/4/8G FIBRE CHANNEL

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16N55-M16UP

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

!

Reset

Console

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108


! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

!

Reset

Console

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

SLOT

1

SLOT

5

SLOT

3

SLOT

7

SLOT

2

SLOT

6

SLOT

4

SLOT

8

!

UCS 5108


! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

!

Reset

Console

UCS B200 M3

! ResetConsole

UCS B200 M3

! ResetConsole

UCS B200 M3

ACI Leaf ACI Leaf ACI Border

LeafACI Border Leaf

UCS

FW

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

Cisco Nexus 2148T

1GE Fabric Extender

STAT

ID 45 46 47 48 1 2 3 4

ADC

ADC

CISCO UCS 6248UP 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

STAT

ID

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

PW

R

SY

S

CO

NS

OL

E

UCS

C240 M3

!

Servers

FW ADC

SEC

VMVM

VM

Adaptive

Security

Appliance

Cisco

ASA 5545-X

1

0

ALARM

VPN

HD1

HD0

BOOT

ACTIVE

PS1

PS0

Adaptive

Security

Appliance

Cisco

ASA 5545-X

1

0

ALARM

VPN

HD1

HD0

BOOT

ACTIVE

PS1

PS0

1.11.1

G~ POWER 2~ POWER 1

1.11.1

G~ POWER 2~ POWER 1

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

VM VM

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

40 Gig

10 Gig

VPC

STS

BCN

ACT

Cisco Nexus 9396PX

1

2

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48

© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public 15

VLAN 111

VLAN 222

VLAN 10

Nexus 9396 Nexus 9396

Nexus 7000

ACI-9396 Leaf

ACI-9336 Spine

Nexus 9396 Nexus 9396

vlan 1011

V

D

C

40G

10G

RR

vlan 2022

172.16.222.222172.16.111.11110.96.126.17

10.222.222.1610.111.111.50

VEM

VSM

HYPERVISOR

VXLAN Overlay

10G

10G

10.96.126.80

99.99.99.0/30

.1

.2

150.150.150.0/30

.2.1

A B

10.9.9.0/30

.1

.2

BRKDCN-2200


Under the Hood - Topo

VM

OS

VM

OS

VXLAN L2

Gateway

VXLAN L2

Gateway

VM-A VM-B

VXLAN L2

Gateway

VM-C

L3 – FW,

SLB

LAN Extension

Tunnel

VM

OS

N7k-1 N7k-2

N9k N9k N9k N9k


Under The Hood - Nexus 7000 - Transport

• interface Ethernet1/1

• no switchport

• ip address 10.9.9.1/8

• ip router eigrp 813

• no shutdown

interface Vlan900

description ACI-VLAN900

no shutdown

bandwidth 80000000

no ip redirects

ip address 99.99.99.1/30

no ipv6 redirects

no ip passive-interface eigrp 813

ip pim sparse-mode

interface Vlan901

description Transit_vlan_901_between_sydney23-

sydney24

no shutdown

ip address 150.150.150.2/30

ip router eigrp 813

no ip passive-interface eigrp 813

ip pim sparse-mode

feature tacacs+

cfs eth distribute

feature pim

feature eigrp

feature udld

feature interface-vlan

feature hsrp

feature lacp

feature dhcp

feature vpc

feature sflow

Nexus 7000 - ABoth N7k’s Transit Networks

Design Details - Nexus 9000


Under the Hood – Nexus 9000

VM

OS

VM

OS

VXLAN L2

Gateway

VXLAN L2

Gateway

VM-A VM-B

VXLAN L2

Gateway

VM-C

L3 – FW,

SLB

LAN Extension

Tunnel

VM

OS

N7k-1 N7k-2

N9k N9k N9k N9k


Under the Hood - Each Device BeginsEnable VXLAN and MP-BGP EVPN Control Plane

feature nv overlay

feature vn-segment-vlan-based

feature bgp

nv overlay evpn

Enable VXLAN

Enable VLAN-based VXLAN (the currently

only mode)

Enable OSPF if it’s chosen to be the

underlay IGP routing protocol

Enable VLAN SVI interfaces if the VTEP

needs to be IP gateway and route for the

VXLAN VLAN IP subnet.

Enable EVPN control plane for VXLAN

feature ospf

feature pim

feature interface-vlan

Other features may need to be enabled

Enable BGP

Enable IP PIM multicast routing in the

underlay network


Under the Hood - Tenant CreationVXLAN – virtual routing / forwarding

vrf context vxlan-n1k-vm-a

vni 22200

rd auto

address-family ipv4 unicast

route-target import 22200:22200

route-target export 22200:22200

route-target both auto evpn

Create a VXLAN Tenant VRF

Specify the Layer-3 VNI for VXLAN routing

within the tenant VRF

Define VRF Route Target and import/export

policies in address-family ipv4 unicast

Define VRF RD (route distinguisher)

vrf context vxlan-n1k-vm-b

vni 22210

rd auto





Example to create a 2nd tenant VRF

following the above steps


Under the Hood - Layer-3 (VNI) Routing – VM(A)Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant

vlan 2220name vrf-L3-vm-avn-segment 22200

interface Vlan2220description vrf-L3-vm-a-routingno shutdownvrf member vxlan-n1k-vm-a

vrf context vxlan-n1k-vm-avni 22200rd autoaddress-family ipv4 unicastroute-target import 22200:22200route-target export 22200:22200route-target both auto evpn

Create the VLAN for the Layer-3 VNI.

One Layer-3 VNI per tenant VRF routing

instance

Create the SVI interface for the Layer-3 VNI

Put this SVI interface into the tenant VRF

context

Associate the Layer-3 VNI with the tenant

VRF routing instance.


Under the Hood - Layer-3 (VNI) Routing – VM(B)Configure Layer-3 VNI per EVPN Tenant VRF Routing Instant

vlan 1110

name vrf-L3-vm-b

vn-segment 22210

interface Vlan1110

description vrf-L3-vm-b-routing

no shutdown

vrf member vxlan-n1k-vm-b

vrf context vxlan-n1k-vm-b

vni 22210

rd auto





Define Layer-3 VNI for a 2nd tenant

following the same steps in the previous

slide


Under the Hood - Layer-2 VXLAN Network Identifier Map VLANs to VXLAN VNIs and Configure their MP-BGP EVPN Parameters

vlan 222

vn-segment 20000

vlan 111

vn-segment 21000

Map VLAN to VXLAN VNI

evpn

vni 20000 l2

rd auto

route-target import auto

route-target export auto

vni 21000 l2

rd auto

route-target import auto

route-target export auto

Under EVPN configuration, define RD

and RT import/export policies for each

Layer-2 VNIs


Under the Hood - Interface SVI – Layer 2Create SVI interface for Layer-2 VNIs for VXLAN routing

interface Vlan111

no shutdown

vrf member vxlan-n1k-vm-a

ip address 10.111.111.1/8

fabric forwarding mode anycast-gateway

interface Vlan222

no shutdown

vrf member vxlan-n1k-vm-b

ip address 10.222.222.1/8


Create SVI interface for a Layer-2 VNI.

Associate it with the tenant VRF.

Enable distributed anycast gateway for this

VLAN/VNI

All VTEPs for this VLAN/VNI should have the

same SVI interface IP address as the

distributed IP gateway.


Under the Hood - Distributed Gateway – Anycast

fabric forwarding anycast-gateway-mac 0000.1111.2222

interface Vlan111

no shutdown

vrf member vxlan-n1k-vm-a

ip address 10.111.111.1/8


Configure virtual IP address

All VTEPs for this VLAN should have the same

virtual IP address

Configure distributed gateway virtual MAC

address

One virtual MAC per VTEP

All VTEPs should have the same virtual MAC

address

Enable distributed gateway for this VLAN


Under the Hood - Network Virtualization Endpoint Configure VXLAN tunnel interface nve1

interface nve1

no shutdown

source-interface loopback0

host-reachability protocol bgp

member vni 20000

suppress-arp

mcast-group 239.1.1.1

member vni 21000

suppress-arp

mcast-group 239.1.1.2

member vni 22200 associate-vrf

member vni 22210 associate-vrf

Specify loopback0 as the source interface

Define BGP as the mechanism for host

reachability advertisement

Add Layer-3 VNIs, one per tenant VRF

Associate tenant VNIs to the tunnel

interface nve1

Define the mcast group on a per-VNI basis

Enable arp suppression on a per-VNI basis


Under the Hood - VXLAN Tunnel Interface Configuration – Cont’d Configure VXLAN tunnel interface nve1

interface loopback 0

ip address 10.111.222.1/32

ip ospf network point-to-point

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

The loopback interface to source VXLAN

tunnels


Under the Hood - BGP – “Yes” in the LANrouter bgp 65535

router-id 10.111.222.1

log-neighbor-changes


address-family l2vpn evpn

neighbor 10.1.2.1 remote-as 65535

update-source loopback0



send-community extended

neighbor 10.1.2.2 remote-as 65535




send-community extended

vrf vxlan-n1k-vm-a


advertise l2vpn evpn

vrf vxlan-n1k-vm-b


advertise l2vpn evpn

Address-family ipv4 unicast for prefix-

based routing

Define MP-BGP neighbors.

Under each neighbor define address-family

ipv4 unicast and l2vpn evpn

Under address-family ipv4 unicast of each

tenant VRF instance, enable advertising

EVPN routes

Send extended community in l2vpn evpn

address-family to distribute EVPN route

attributes

Address-family l2vpn evpn for evpn host

routes


Under the Hood - Route Reflectorrouter bgp 65535

router-id 10.1.2.1

log-neighbor-changes



retain route-target all

template peer vtep-peer

remote-as 65535



send-community both

route-reflector-client


send-community both

route-reflector-client

neighbor 10.111.222.1

inherit peer vtep-peer

neighbor 10.1.2.12

inherit peer vtep-peer

Address-family ipv4 unicast for prefix-

based routing

iBGP RR client peer template

Send both standard and extended

community in address-family l2vpn evpn

Send both standard and extended

community in address-family ipv4 unicast

Address-family l2vpn evpn for EVPN vxlan

host routes

Retain route-targets attributes

Design Details - Nexus 1000


Cisco Nexus 1000V Architecture

Hypervisor Hypervisor Hypervisor

VEM-NVEM-1 VEM-2

VSM: Virtual Supervisor Module

VEM: Virtual Ethernet Module Server

Admin

NX-OS

Data Plane

VSM-1 (active)

VSM-2 (standby)

Virtual ApplianceNX-OS

Control PlaneNetwork

Admin

Modular Switch

…

Linecard-N

Supervisor-1 (Active)

Supervisor-2 (StandBy)

Linecard-1

Linecard-2

Ba

ck P

lan

e

BRKDCN-2200 32


Under the Hood - Nexus 1000v

VM

OS

VM

OS

VXLAN L2

Gateway

VXLAN L2

Gateway

N7k-1 N7k-2

VM-A VM-B

VXLAN L2

Gateway

10.222.222.50

111.111.111.110.222.222.1

VM-C 10.222.222.49

L3 – FW,

SLB

LAN Extension

Tunnel

111.111.111.x

VM

OS


Under the Hood - Nexus 1000v transport

n1kv-wayne# sh run

version 5.2(1)SV3(1.5a)

hostname n1kv-wayne

port-profile type ethernet UPLINK

switchport mode trunk

switchport trunk allowed vlan 1-2,100-300

channel-group auto mode on mac-pinning

no shutdown

system vlan 1-2

state enabled

vmware port-group

port-profile type vethernet L3-Control

switchport mode access

switchport access vlan 1

no shutdown

capability l3control

system vlan 1

state enabled

vmware port-group

vrf context management

ip route 0.0.0.0/0 10.96.126.254

vlan 1-2,100-300

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

shutdown

description Port-group created for Nexus 1000V internal usage. Do not use.

state enabled

vmware port-group

port-profile type vethernet Unused_Or_Quarantine_Veth

shutdown

description Port-group created for Nexus 1000V internal usage. Do not use.

state enabled

vmware port-group

interface Vethernet1

inherit port-profile L3-Control

description VMware VMkernel, vmk2

vmware dvport 100 dvswitch uuid "75 3e 37 50 a5 6b ef f6-85 60 6a 7a 7f b6

3d"

vmware vm mac 0050.5671.47DA

interface Vethernet3

inherit port-profile vm-222

description Windows-7-222, Network Adapter 1

vmware dvport 256 dvswitch uuid "75 3e 37 50 a5 6b ef f6-85 60 6a 7a 7f b6

3d"

vmware vm mac 0050.56B7.0108

port-profile type vethernet EVPN-VXLAN




Under the Hood - VXLAN Forwarding Basics - VSM

VEM 1 VEM 2

Forwarding mechanisms similar to Layer 2 bridge: Flood & Learn

VEM learns VM’s Source (MAC, Host VXLAN IP) tuple

Broadcast, Multicast, and Unknown Unicast Traffic

VM broadcast & unknown unicast traffic are sent as multicast

Unicast Traffic

Unicast packets are encapsulated and sent directly (not via multicast) to destination host VXLAN IP (Destination VEM)

VM VM VM VM


Under the Hood - VM Host - VXLAN Topo

Guest Machine(s)

configured for setup


L3 – N9k Enhanced VXLAN – VSM

VLAN CLI Model

vlan 222name – n1k

interface vlan 222ip address 10.222.222.1ip router eigrp 22

interface Ethernet3/2switchportswitchport mode trunkrate-mode dedicated forcechannel-group 222 mode activeno shutdown

VSM config

feature segmentation

segment mode unicast-only

port-profile type vethernet vxlan-n1k


vmware port-group



capability vxlan

no shutdown

system vlan 1

state enabled

37

Normal SVI’s

BRKDCN-2200

• VMkernel interface acts as VTEP

• VSM Control Mode should be L3

• Bridge domain is configured as Unicast or

Unicast Mac Distribution

B+U – no “M”

Under the Hood -


Under the Hood - VMKernel

port-profile type vethernet vxlan-n1k


vmware port-group



capability vxlan

no shutdown

system vlan 1

state enabled


Under the Hood – VSM Bridge Domain

port-profile type vethernet bd-22222

switchport access bridge-domain BD-vxl

no shutdown

state enabled

vmware port-group

port-profile type vethernet vmk-l3-vxlan-vtep



capability vxlan

no shutdown


state enabled

vmware port-group


Under the Hood - Port Profile Attachment

N1K - DVS


Under the Hood – VTEP status

vsm-vxlan# show bridge-domain bd-22222

Bridge-domain bd-22222 (2 ports in all)

Segment ID: 22222 (Manual/Active)

Mode: Unicast-only (override)

MAC Distribution: Disable (override)

Group IP: NULL

State: UP Mac learning: Enabled

Veth4, Veth18

vsm-vxlan# show bridge-domain bd-22222 vteps

Bridge-domain: bd-22222

VTEP Table Version: 21

Port Module VTEP-IP Address VTEP-Flags

---------------------------------------------------------------------------

Veth1 3 10.111.111.49 (D) <---Designated VTEP (vmk)

Veth2 4 10.111.111.50 (D)

41BRKDCN-2200

Avoid Resume Generating Event(s)


Best Practice(s)

43BRKDCN-2200

What Should We Do?

• Optimization

• Decision Trees

Hit the

EASY BUTTON

LCM

• Backups

• High Availability Options

• Software Repository

Success

Deployment

Virtual Switch Update Manager

Life Cycle Management – VSM / VEM

Performance & Scalability

What Should We Do?

What’s the desired outcome?

Enterprise Architecture Framework – Network, Security, Server, Virtualization

P & S

• HW Limits

• SW Limits


Performance & Scalability

• ESX – 5.2(1)SV3(1.2)

• 256 VEMs, 12K vEth count

• VxLAN 2.0 (BGP Control Plane)

• VxLAN UDP Port Configurable

• N1K Virtual Switch Update Manager

• Distributed NetFlow

• IGMP Multicast Offload (1k Groups)

• BPDU Guard & Storm Control

• Cisco TrustSec, IPv6 Enhancements

• ESX – 4.2(1)SV2(2.2)

• Dynamic Fabric Automation Leaf

• VDP – VSI Discovery Protocol

• Universal Licensing

• Hyper-V – 5.2(1)SM3(1.1)

• VxLAN 1.0 & HVN

• Hyper-V – 5.2(1)SM1(5.2a)

• SCVMM 2012 SP1 & R2

• Windows Server 2012 & R2

• VSG VM and Custom Attributes

• Universal Licensing

• KVM – 5.2(1)SK3(2.1)

• IceHouse

• RHEL-OSP – OpenStack Platform Inst

• VxLAN GW

• pVLAN

• UUFB blocking44

BP

BRKDCN-2200


Life Cycle Management - VSM Control Modes

• L3 Mode

• This is not routing

• L3 is the recommended & default• Easier to troubleshoot

• Cross Firewalls & L3 boundaries

• Requires an IP address be assigned to the VEM (vmk)

• Uses UDP 4785 for both source and destination

• Sourced from mgmt0 by default

• L2 mode (Legacy)

• Requires L2 connectivity through control0 interface to all VEM modules

• Deprecated but supported on ESX

• Not supported with Hyper-V or KVM

45BRKDCN-2200

BP


Life Cycle Management - VSM vMotion

• Manual vMotion/Live Migration is supported

• VMware DRS is NOT recommended for Primary & Secondary VSMs

• Aggressive DRS could lead to excessive VSM-VEM heartbeat packet drops

• Best practice to keep Primary and Secondary VSM outside DRS control

• Use anti-affinity rules where possible

• FT is not supported

46

BP

BRKDCN-2200


Life Cycle Management - VSM Backups

• A running-config is not enough to restore due to PSS

• VSM on ESXi / HyperV

• Clone to a template

• Restore from an older template + running-config

• Both VSMs must be powered down

• VSM on Nexus 1110

• Export a VSM to a file

• Import the saved VSM to restore

• VSM on ESXi Snapshots

• Not officially supported

• I/O latency cost associated with expanding the differential file

47

BP

BRKDCN-2200


Life Cycle Management - VSM Interfaces

• Control

• VSM-VSM HA Heartbeats

• VSM-VEM Heartbeats

• VSM-VSM Synchronization

• BGP Control Plane

• Packet

• CDP, IGMP*, SNMP

• Layer3 Mode

• Collapsed ctrl0 & pkt into mgmt0

• VSM-VEM communication on mgmt0

• Dedicated Control:svs mode L3 interface [control | mgmt0]

• Management

• SSH console access

• SNMP, HTTP, XML

• vCenter Communication

• HA Heartbeat Backup

• Interface Order is always the same!

VSM-Peth0: control

eth1: mgmt0

eth2: packet

48

BP

BRKDCN-2200


Life Cycle Management - VEM Deployment

• L3 control requires a VMKernel NIC on N1K DVS

• We need an L3 interface to forward control traffic

• 200/100/10ms latency between VSM & VEM

• Recommend using the ESXi management VMKernel NIC

• Migrate management vmk behind VEM

• Doesn’t require static routes on ESXi hosts

• Put additional vmks on different subnets (vMotion / Storage)

• UCS “Dynamic vNICs” in Service-Profiles

• VEM and VM-FEX are mutually exclusive

49BRKDCN-2200

BP


VEM Deployment – VMKs on same subnet

• VMware uses a single TCP/IP stack for all VMK interfaces

• Don’t use multiple VMKs on the same subnet on different virtual switches

• No way to pin traffic to an uplink interface.

• One interface gets picked for all traffic on that subnet

• VMware KB article 2010877

• Only one default gateway per host

VMware ESX

VEM

VMK1

192.168.10.100

VMK0

192.168.10.200

vSwitch

50

BP

BRKDCN-2200


VEM - Port-Profiles Secret Sauce

port-profile type ethernet uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 10,119

channel-group auto mode on mac-pinning

no shutdown

system vlan 119

state enabled

vmnic0 Eth3/1

vmnic1 Eth3/2

port-profile type vethernet vmk-l3


vmware port-group



capability vxlan

no shutdown

system vlan 119

state enabled

VM1

VMK1

VM2port-profile type vethernet vm-vlan10

vmware port-group



no shutdown

state enabled

PO1

BP

vEthernet PP (default)

-Virtual Interfaces (vEthernet x)

-Typically Access Ports or Bridge Domains

-Configuration: VLAN, ACLs, VxLAN, QoS

Ethernet PP

-Physical Interfaces (Ethernet x/y)

-Typically Trunk Ports

-Configuration: Port-Channel, ACLs, QoS


VSUM – Virtual Switch Update Manager

• Install, Migrate, Upgrade, Monitor Nexus 1000V and ACI AVS

• Standalone VM

• Nexus 1000V Binaries are Self-Contained

• Integrated in vSphere Web Client through Plugin

• VMware only today

• Single instance manages all N1k on a vCenter

• Manages existing N1k DVS

No

Charge

BP


VSUM – Plugin IconBP


VSUM – Installing Nexus 1000V VSM

54

1 2

3

BP

BRKDCN-2200


Upgrades - Deployment

• First always read and follow the upgrade guides

• Order matters: VSM then VEM

• Take a backup of the VSMs

• On ESXi use the clone to template option (Powered Down)

• On Nexus 1110s / Cloud Services Platform use the export command

• Backup the running-config

• Generate a tech-support before the upgrade

• If something goes wrong STOP and call TAC

• Use a maintenance window

• VEM upgrades require ESXi hosts to be in Maintenance Mode

• Use N1k Upgrade Utility Matrix to Plan a combined N1k+vSphere Upgrade

BP

55BRKDCN-2200

http://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/nexus1000/upgrade/n1kvmatrix.html

Design Details – Application

BRKDCN-2200

CentricInfrastructure


ACI Relationship Map

1

3

2 7 4 569

8


End-Points end EPG membership

Device connected to network directly or indirectly

Has address (identity), location, attributes (version, patch level)

Can be physical or virtual or container

• Examples:

• End Point Group (EPG) membership defined by:• Ingress physical port (leaf or FEX)

• Ingress logical port (VM port group)

• VLAN ID

• VXLAN (VNID)

• IP address

• IP Prefix/Subnet

• VM-based attributes

• NVGRE (VSID) (future)

• Layer 4 ports (future)

Server

Virtual Machines & Containers

Storage

Client

58BRKDCN-2200


ACI – Segmentation

Micro-segmentation


• Background• 111.111.111.10 and 111.111.111.11 in CL-VXLAN

• Create CL-VXLAN(useg)

• Put 111.111.111.10/32 in EPG VM-1(useg)

• Both VM – talk to GW + Each other !

LEAF 222

1/17

VLAN 111

111.111.111.11

BaseCL-

VXLAN

Can talk to each other as they are in same EPG

111.111.111.10

BaseCL-VXLAN

BD1

LEAF 222

1/7

VLAN 222

LEAF 222

1/17

VLAN 111

111.111.111.11

CL-VXLAN

111.111.111.10

CL-

VXLAN(useg)

BD1

LEAF 222

1/7

VLAN 222

Still can talk to each other

Configure CL-VXLAN(useg)

bypasses IP classification

BRKDCN-2200 60

Case Study Deployment Scenario's

BRKDCN-2200


Case Study Deployment: Basic Tunneling

• VSM is a Virtual Machine

• Control plane for the Nexus 1000V switch

• VEM packet forwarding not impacted by reloads

• VSM HA pair distributed across multiple host

• Responsible for:

• Programming and Managing Virtual Ethernet Modules (VEM)

• Communicating with Management Applications (vCenter, SCVMM, Horizon Dashboard, etc.)

62BRKDCN-2200

Hypervisor

VEM

VM VMVM

VSM

#1


SPINE SPINE

TPA

LEAF

3

LEAF

SVC Block

I-NETINTEXT I-NET INT EXTDFW

FI

FI

SCE

SAP MSFT RAC RHEL

NGFWNGFWFI

FI

SCE

SAP MSFT RAC RHEL

N2kN2kN2kN2k

SERVER ACCESS

N2kN2kN2kN2k

SERVER ACCESS

UNIFIED COMPUTE SYSTEM UNIFIED COMPUTE SYSTEM

Case Study - Deployment

N2k N2k N2k N2k

#2


DMZ

Case Study - ACI LEVEL

Data Farm

Data Farm

OSPF

VM VMVM

#3

Roadmap


Now is the future• Migrate Customers from Nexus 1010/1010-X/1110-S/1110-X

• Dedicated Cisco Cloud Services Platform appliance ( CSP 2100 )

• Preparation for Nexus 1000 release 3 – BGP control plane interoperability with Nexus 9000

• Whitepaper to follow – design guidance on VM scale and extended attribute parity

• Look @ ACI – you just might “love it”

66BRKDCN-2200


Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.

67BRKDCN-2200

CiscoLive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/Online

http://ciscolive.com/us

http://ciscolive.com/us


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Lunch & Learn

• Meet the Engineer 1:1 meetings

• Related sessions

68BRKDCN-2200

Thank you


R&S Related Cisco Education OfferingsCourse Description Cisco Certification

CCIE R&S Advanced Workshops (CIERS-1 &

CIERS-2) plus

Self Assessments, Workbooks & Labs

Expert level trainings including: instructor led workshops, self

assessments, practice labs and CCIE Lab Builder to prepare candidates

for the CCIE R&S practical exam.

CCIE® Routing & Switching

• Implementing Cisco IP Routing v2.0

• Implementing Cisco IP Switched

Networks V2.0

• Troubleshooting and Maintaining

Cisco IP Networks v2.0

Professional level instructor led trainings to prepare candidates for the

CCNP R&S exams (ROUTE, SWITCH and TSHOOT). Also available in

self study eLearning formats with Cisco Learning Labs.

CCNP® Routing & Switching

Interconnecting Cisco Networking Devices:

Part 2 (or combined)

Configure, implement and troubleshoot local and wide-area IPv4 and IPv6

networks. Also available in self study eLearning format with Cisco Learning

Lab.

CCNA® Routing & Switching

Interconnecting Cisco Networking Devices:

Part 1

Installation, configuration, and basic support of a branch network. Also

available in self study eLearning format with Cisco Learning Lab.

CCENT® Routing & Switching

71

For more details, please visit: http://learningnetwork.cisco.com

Questions? Visit the Learning@Cisco Booth or contact [email protected]

BRKDCN-2200

http://learningnetwork.cisco.com/

mailto:[email protected]


Design Cisco Education OfferingsCourse Description Cisco Certification

Designing Cisco Network Service Architectures

(ARCH) Version 3.0

Provides learner with the ability to perform conceptual, intermediate, and

detailed design of a network infrastructure that supports desired capacity,

performance, availability required for converged Enterprise network

services and applications.

CCDP® (Design Professional)

(Available Now)

Designing for Cisco Internetwork Solutions

(DESGN) Version 3.0

Instructor led training focused on fundamental design methodologies used

to determine requirements for network performance, security, voice, and

wireless solutions. Prepares candidates for the CCDA certification exam.

CCDA® (Design Associate)

(Available Now)



72BRKDCN-2200




Data Center / Virtualization Cisco Education OfferingsCourse Description Cisco Certification

Introducing Cisco Data Center Networking (DCICN);

Introducing Cisco Data Center Technologies (DCICT)

Learn basic data center technologies and skills to build a

data center infrastructure.

CCNA® Data Center

Implementing Cisco Data Center Unified Fabric (DCUFI);

Implementing Cisco Data Center Unified Computing (DCUCI)

Designing Cisco Data Center Unified Computing (DCUDC)

Designing Cisco Data Center Unified Fabric (DCUFD)

Troubleshooting Cisco Data Center Unified Computing

(DCUCT)

Troubleshooting Cisco Data Center Unified Fabric (DCUFT)

Obtain professional level skills to design, configure,

implement, troubleshoot data center network infrastructure.

CCNP® Data Center

Product Training Portfolio: DCNMM, DCAC9K, DCINX9K,

DCMDS, DCUCS, DCNX1K, DCNX5K, DCNX7K

Gain hands-on skills using Cisco solutions to configure,

deploy, manage and troubleshoot unified computing, policy-

driven and virtualized data center network infrastructure.

Designing the FlexPod® Solution (FPDESIGN);

Implementing and Administering the FlexPod® Solution

(FPIMPADM)

Learn how to design, implement and administer FlexPod

solutions

Cisco and NetApp Certified

FlexPod® Specialist

73



BRKDCN-2200




Network Programmability Cisco Education OfferingsCourse Description Cisco Certification

Integrating Business Applications with Network

Programmability (NIPBA);

Integrating Business Applications with Network

Programmability for Cisco ACI (NPIBAACI)

Learn networking concepts, and how to deploy and troubleshoot

programmable network architectures with these self-paced courses.

Cisco Business Application

Engineer Specialist Certification

Developing with Cisco Network Programmability

(NPDEV);

Developing with Cisco Network Programmability

for Cisco ACI (NPDEVACI)

Learn how to build applications for network environments and effectively

bridge the gap between IT professionals and software developers.

Cisco Network Programmability

Developer Specialist Certification

Designing with Cisco Network Programmability

(NPDES);

Designing with Cisco Network Programmability

for Cisco ACI (NPDESACI)

Learn how to expand your skill set from traditional IT infrastructure to

application integration through programmability.


Design Specialist Certification

Implementing Cisco Network Programmability

(NPENG);

Implementing Cisco Network Programmability

for Cisco ACI (NPENGACI)

Learn how to implement and troubleshoot open IT infrastructure

technologies.


Engineer Specialist Certification



74BRKDCN-2200




Cloud Cisco Education OfferingsCourse Description Cisco Certification

Understanding Cloud Fundamentals

(CLDFND)

Learn how to perform foundational tasks related to Cloud computing, and the essentials

of Cloud infrastructureCCNA Cloud

Introducing Cloud Administration

(CLDADM)

Learn the essentials of Cloud administration and operations, including how to provision,

manage, monitor, report and remediate.

Implementing and Troubleshooting the

Cisco Cloud Infrastructure (CLDINF)

Learn how to implement and troubleshoot Cisco Cloud infrastructure: compute,

network, storage.

CCNP Cloud

Designing the Cisco Cloud (CLDDES)*Learn how to design private and hybrid Clouds including infrastructure, automation,

security and virtual network services

Automating the Cisco Enterprise Cloud

(CLDAUT)*

Learn how to automate Cloud deployments – provisioning IaaS (private, private with

network automation and hybrid) and applications, life cycle management

Building the Cisco Cloud with Application

Centric Infrastructure (CLDACI)*

Learn how to build Cloud infrastructures based on Cisco Application Centric

Infrastructure, including design, implementation and automation

UCS Director Foundation (UCSDF)Learn how to manage physical and virtual infrastructure using orchestration and

automation functions of UCS Director.

75

* Available Q2CY2016



BRKDCN-2200



Documents

VXLAN - Bringing Hypervisord2zmdbbm9feqrf.cloudfront.net/2016/usa/pdf/BRKDCN-2200.pdf · VXLAN - Bringing Hypervisor & Nexus Together Wayne Davis –Technical Solutions Architect