Everyone in this room is a GENIUS

What are Best Practices ?

Learning from Others Mistakes

4

Learning from your mistakes makes you

SMARTLearning from others mistakes makes you

GENIUS

vPC Best Practices and Design on NXOS

Nazim Khan, CCIE#39502 (DC/SP)

Technical Marketing Engineer, Data Center Group

BRKDCT-2378

Session Goals

• Best Practices and Designs for vPC – virtual port-channel

• Nexus 2000 (FEX) will only be addressed from vPC standpoint

• Fabricpath / vPC+ Overview

• vPC with FCOE

• vPC with VXLAN

• vPC with ACI

vPC : Get it Right the very First time

Session Non-Goals

• vPC troubleshooting

• Details of vPC+

• Details of Fabricpath, FCoE, ACI and VXLAN

Related Sessions at Cisco Live San Diego

Session Id Session Name

BRKDCT-2404 VXLAN deployment models - A practical perspective

BRKDCT-2081 Cisco FabricPath Technology and Design

BRKDCT-2458Nexus 9000/7000/6000/5000 Operations and

Maintenance Best Practices

BRKACI-2601 Real World ACI Deployment and Migration

Agenda

• Feature Overview

• Configuration Best Practices

• Design Best Practices

• vPC Operations and Upgrade

• vPC with Fabric Technologies

• Scalability

• Reference Material

Data Center Technology Evolution

FabricPath with vPC+

2010

2009

VPC

2008

STP2013-2014

MPLS, OTV,

LISP

VXLAN

MPLS, OTV,

LISP

2014-2015

ACI

2010

FEX with vPC

Why vPC in 2015 ?

12

vPC is Foundation

Role of vPC in the Evolution of Data Center

• vPC launched in 2009

• Deployed by almost 95% of Nexus customers

• Used to redundantly connect network entities at the

edge of the Fabric

− Dual-homed servers (bare metal, blades, etc.)

− Network services (Firewalls, Load Balancers, etc.)

Unified Fabric

Agenda

• Feature Overview− Concepts and Benefits− Terminology

vPC Feature OverviewvPC Concept & Benefits

S1 S2

S3

• No Blocked Ports, More Usable Bandwidth, Load Sharing

• Fast Convergence

STP

S2S1

S3

vPC Logical Topology

S3

S1 S2

vPC Physical Topology

Feature OverviewvPC Terminology

Layer 3 Cloud

vPC Member PortvPC

Orphan Device

Orphan Port

vPC Peer

CFS

vPC Domain

Peer-Link

vPC PeerKeepalive Link

S1

S3

S2

SW3 SW4

vPC1 vPC2

vPC_PLink

vPC Peer-keepalive

vPC Failure Scenario vPC Peer-Keepalive Link up & vPC Peer-Link down

P S

Keepalive Heartbeat

Secondary vPCS

P Primary vPC

Suspend secondary

vPC Member Ports

vPC peer-link failure (link loss):

• vPC peer-keepalive up

• Status of other vPC peer known

• Both peers Active

• Secondary vPC peer disables all vPC’s

• Traffic from vPC primary.

• Orphan devices connected to secondary peer will be isolated

For YourReference

vPC Failure Scenario – Dual ActivevPC Peer-Keepalive down followed by vPC Peer-Link down

P S

Secondary vPCS

P Primary vPC

P

SW3 SW4

vPC1 vPC2

vPC_PLink

vPC Peer-keepalive

Traffic Loss / Uncertain Traffic

Behavior

1. vPC peer-keepalive DOWN

2. vPC peer-link DOWN

3. DUAL-ACTIVE or SPLIT BRAIN

• vPC primary peer remains primary and secondary peer becomes operational primary role

• Result in traffic loss / uncertain traffic behavior

• When links are restored, the operational primary (former secondary) keeps the primary role & former primary becomes operational secondary

For YourReference

Agenda

• vPC Configuration Best Practices− Building a vPC domain− Domain-ID− Peer-Link− Peer-Keepalive Link− Spanning-Tree− Peer-switch− Private VLAN (PVLAN)− Auto-recovery− Object tracking

vPC Configuration Best PracticesBuilding a vPC domain – Configuration Steps

CFS

1. Define domains

2. Establish Peer Keepalive connectivity

3. Create a Peer link

4. Create vPCs

5. Make Sure Configurations are Consistent

(Order does Matter!)

S1 S2

S3

vPC Configuration Best PracticesvPC Domain-ID

vPC Domain 10

vPC Domain 20

• The vPC peer devices use the vPC domain ID to

automatically assign a unique vPC system MAC

address

• You MUST use unique Domain id’s for all vPC

pairs defined in a contiguous layer 2 domain

! Configure the vPC Domain ID – It should be unique within the layer 2

domain

NX-1(config)# vpc domain 20

! Check the vPC system MAC address

NX-1# show vpc role

<snip>

vPC system-mac : 00:23:04:ee:be:14

S1 S2

S3 S4

S5

vPC Configuration Best PracticesvPC Peer-Link

• vPC Peer-link should be a point-to-point connection

• Peer-Link member ports can be 10/40/100GE interfaces

• Peer-Link bandwidth should be designed as per the vPC

• vPC imposes the rule that peer-link should never be blocking

S1 S2

S3

S1 S2

S3

vPC Configuration Best PracticesvPC Peer-Keepalive link

Preference Nexus 7X00 /

9500 series

Nexus 9300 /6000 /

5X00 / 3X00 series

1 Dedicated link(s)

(1GE/10GE LC)

mgmt0 interface

2 mgmt0 interface Dedicated link(s)

(1GE/10GE LC)

3 L3 infrastructure L3 infrastructure

Recommendations

(in order of

preference):

vPC Configuration Best PracticesvPC Peer-Keepalive link – Dual Supervisors

Standby Management Interface

Active Management Interface

vPC1 vPC2

vPC_PL

Management Network

Management Switch

vPC_PKLvPC_PKL• When using dual supervisors and mgmt0 interfaces

to carry the vPC peer-keepalive, DO NOT connect

them back to back between the two switches

• Only one management port will be active a given point

in time and a supervisor switchover may break keep-

alive connectivity

• Use the management interface when you have an out-

of-band management network (management switch in

between)

For YourReference

vPC Configuration Best PracticesSpanning Tree (STP)

• All switches in Layer 2 domain should run either Rapid-PVST+ or MST

• Do not disable spanning-tree protocol for any VLAN

• Always define the vPC domain as STP root for all VLAN in that domain

STP is running to manage

loops outside of vPC domain,

or before initial vPC

configuration !

S1 S2

S4S3

S5

vPC Configuration Best PracticesvPC Peer-Gateway

• Allows a vPC switch to act as the active

gateway for packets addressed to the peer

router MAC

• Keeps forwarding of traffic local to the vPC

node and avoids use of the peer-link

• Allows Interoperability with features of some

NAS or load-balancer devices

N7k(config-vpc-domain)# peer-gateway

S1 S2

S4S3

vPC Configuration Best PracticesvPC Peer-switch

Without Peer-switch

• STP for vPCs controlled by vPC primary.

• vPC primary send BPDU’s on STP designated ports

• vPC secondary device proxies BPDU’s to primary

With Peer-switch

• Peer-Switch makes the vPC peer devices to appear as a single STP root

• BPDUs processed by the logical STP root formed by the 2 vPC peer devices

N7k(config-vpc-domain)# peer-switch

Primary

vPC

Secondary

vPC

BPDUs

Primary

vPC

Secondary

vPC

vPC Configuration Best PracticesPVLAN on vPC

• PVLAN configuration across both VPC switches should be identical

• PVLAN configuration not supported on Peer-Link

• Type-1 Compatibility Check

• Port mode is a type-1 check

• vPC leg brought down if PVLAN port mode different on vPC legs

• Type-2 Compatibility Check

• PVLAN will bring down mismatched tuple

S1 S2

vPC Primary vPC Secondary

P P

PVLAN-

PROMISC

(3500, 3501)

PVLAN-

PROMISC

(3500, 3501)

C

Community

VLAN

Note : This feature is currently not supported on N9X00

Pvlan Isolated

trunk

vPC Configuration Best PracticesPVLAN VPC type 1 Consistency Check

S1 S2

vPC Primary vPC Secondary

P PS1 S2

vPC Primary vPC Secondary

I I

Pvlan

Promiscuous

trunk

Type 1

Consistency

Failure

S1 S2

vPC Primary vPC Secondary

TI

S3

S3S3

vPC Configuration Best PracticesPVLAN VPC type 2 Consistency Check

S1 S2

vPC Primary vPC Secondary

P PS1 S2

vPC Primary vPC Secondary

I I

Type 2

Consistency

Failure

S1 S2

vPC Primary vPC Secondary

I

S3

S3S3

I

PVLAN-

PROMISC

(10, 201)

PVLAN-

PROMISC

(10, 201)

Secondary

Trunk (2,31)

(3,30), (4,100)

Secondary

Trunk (2,31)

(3,30), (4,100)

Secondary

Trunk (3,31)

(2,30), (4,100)

Secondary

Trunk (2,31)

(3,30), (4,100)

vPC Configuration Best PracticesvPC auto-recovery

1. vPC peer-link down : S2 - secondary shuts all its vPC member ports

2. S1 down : vPC peer-keepalive link down : S2 receives no keepalives

3. After 3 keepalive timeouts, S2 changes role and brings up its vPCvPC Primary

vPC Secondary

P

S

P

Operational

Primary

S2S1

S3

P S

S1 S2

S3

S1 S2

S3

P S

vPC Configuration Best PracticesvPC auto-recovery

Auto-recovery addresses two cases of single switch behavior

• Peer-link fails and after a while primary switch (or keepalive link) fails

• Both VPC peers are reloaded and only one comes back up

How it works

• If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives will

trigger auto-recovery

• After reload (role is ‘none established’) auto-recovery timer (240 sec) expires while

peer-link and peer-keepalive still down, autorecovery kicks in

• Switch assumes primary role

• VPCs are brought up bypassing consistency checks

For YourReference

Nexus(config)# vpc domain 1

Nexus(config-vpc-domain)# auto-recovery

vPC Configuration Best PracticesWhy Object-Tracking ?

Primary Secondary

• Modules hosting peer-link and uplink fail on

the vPC primary

• Peer-Link is down and vPC Secondary shut all its vPC

• Auto-Recovery does not kick in as peer-keepalive link is active

• Traffic is black holed

S1 S2

S3

S5S4

vPC Configuration Best PracticesObject-tracking

S1 S2

• Object Tracking triggered when the track object goes down

• vPC object tracking, tracks both peer-link and uplinks in a list of Boolean OR

• Traffic forwarded over the remaining vPC peer! Track the vpc peer link

track 1 interface port-channel11 line-protocol

! Track the uplinks

track 2 interface Ethernet1/1 line-protocol

track 3 interface Ethernet1/2 line-protocol

! Combine all tracked objects into one.

! “OR” means if ALL objects are down, this object will go down

track 10 list boolean OR

object 1

object 2

object 3

! If object 10 goes down on the primary vPC peer,

! system will switch over to other vPC peer and disable all local vPCs

vpc domain 1

track 10

• Suspends the vPCs on the impaired device

S4S5

S3

Agenda

• vPC Design Best Practices

− Mixed Hardware across vPC Peers

− FHRP with vPC

− Hybrid topology (vPC and non-vPC)

− vPC and Network Services

− vPC Fex Supported Topologies

− Physical port vPC

− vPC as Data Center Interconnect (DCI)

− Dynamic Routing over VPC

− vPC and Multicast

Design Best PracticesMixed Hardware across vPC Peers : Line Cards

Always use identical line cards on either sides of the peer link and VPC legs !

vPC Peer-link

S1 S2

vPC Primary vPC Secondary

M2M1

vPC

Examples

vPC Peer-link

S1

N7000

S2

N7700

vPC Primary vPC Secondary

F3

vPC

F3

F2E F2E

Design Best Practices

X Y vPC

N9K-X9636PQ N9K-X9432PQ

N9K-X9564PX N9K-X9464PX

N9K-X9564TX N9K-X9464TX

N9K-X9536PQ N9K-X9736PQ

vPC Peer-link

S1

N9500S2

N9500

vPC Primary vPC Secondary

Y

vPC

X

X Y

Mixed Hardware across vPC Peers : Nexus 9500

Design Best PracticesMixed Hardware across vPC Peers : Chassis & Supervisors

• N7000 and N7700 in same vPC Construct -Supported

• VDC type should match on both peer device

• vPC peers can have mixed SUP version* (SUP1, SUP2, SUP2E)

• N5500 and N5600 in same vPC Construct –Not Supported

*Recommended only for short period such as migration

N5500 N5600

vPC Primary vPC Secondary

S1

N7000

S2

N7700

vPC Primary vPC Secondary

S1 S2

FHRP with vPCHSRP / VRRP/ GLBP Active/Active

FHRP

“Standby”:

Active for

shared L3 MAC

FHRP

“Active”:

Active for

shared L3 MAC

• FHRP in Active/Active mode with vPC

• No requirement for aggressive FHRP timers

• Best Practice : Use default FHRP timers

S1 S2

S3 S4

P

L3L2

OSPF/EIGRP

Primary

vPC

Secondary

vPC

OSPF/EIGRP

VLAN 99

FHRP with vPCBackup Routing Path

• Point-to-point dynamic routing protocol adjacency between the vPC peers to establish a L3 backup path to the core through PL in case of uplinks failure

• Define SVIs associated with FHRP as routing passive-interfaces in order to avoid routing adjacencies over vPC peer-link

• A single point-to-point VLAN/SVI (aka transit vlan) will suffice to establish a L3 neighbor

• Alternatively, use an L3 point-to-point link between the vPC peers to establish a L3 backup path

Routing Protocol Peer

P

Use one transit vlan to establish L3 routing

backup path over the vPC peerlink in case L3

uplinks were to fail, all other SVIs can use

passive-interfaces

S2S1

S4S3

S5

P

P

P

Hybrid topology (vPC and non-vPC)

S1 S2

S3 S4

vPC Primary vPC Secondary

vPC1

Bridge Priority

VLAN 1 4K

VLAN 2 8K

Bridge Priority

VLAN 1 8K

VLAN 2 4K

STP Root

VLAN 1STP Root

VLAN 2

STP Root

VLAN 1

VLAN 2

VLAN 1

(blocked)

VLAN 2

(blocked)

• Supports hybrid topology where vPC and non-vPC are connected to the same vPC domain

• Need additional configuration parameters : spanning-tree pseudo-information

• If previously configured global spanning tree parameters and subsequently configure spanning tree pseudo information parameters, then pseudo information parameters take precedence over the global parameters.

peer-switch

Design Best PracticesASA Cluster

ASA Cluster Mode

• Use unique vPC for ASA Cluster Data Links to vPC domain

• Use vPC per ASA device for Cluster Control Link (CCL) to vPC domain

• Leverage peer-switch configuration

Cluster

Control Link

Cluster

Data Link

Nexus 2000 (FEX) Straight-Through Deployment with VPC

• Port-channel connectivity from the server

• Two Nexus switches bundled into a vPC pair

• Suited for servers with Dual NIC and capable of running Port-Channel

Fex 100

Fabric Links

Fex 101

VPC

HIF HIF

S1 S2

Nexus 2000 (FEX)Active-Active Deployment with VPC

Fabric Links

HIFHIF

Fex 100Fex 101

S1 S2

• Fabric Extender connected to two

Nexus 5X00 / 6000

• Suited for servers with Single NIC or

Dual NIC not having port-channel

capability.

• Scale implications of less FEX per

system and less VPC

Note : • This design is currently not supported on Nexus 9X00

• Nexus 7X00 will support this from release 7.2

Nexus 2000 (FEX) - Enhanced VPC

Fabric Links

Fex 100 Fex 101

S1 S2• Port-channel connectivity to dual-

homed FEXs

• From the server perspective a single

access switch with port-channel

support – each line card supported by

redundant supervisors

• Ideal design for a combination of

single NIC and Dual NIC servers with

port-channel capability

• Scale implications of less FEX per

system and less VPC

Note :

This design is currently not supported on N7000 / N7700 and N9X00

HIFHIF

Physical Port vPC

• vPC configuration on a physical Layer 2 port as opposed to a port-channel

• Front panel ports and FEX ports connected to F2/F2e/F3 only

• Improves scaling as separate PC interface not created for single-link VPC leg

• Key benefit: more than 1000 host facing VPCs with FEX

vPC domain

FEX101 FEX102

e101/1/1 e102/1/1Po1

Port-channel vPC

VPC1 VPC1Po1

vPC domain

FEX101 FEX102

e101/1/1 e102/1/1

Physical port vPC

VPC1 VPC1

Q2 CY15

NX-OS 7.2

(for FEX, scale and

F3 support)

interface e101/1/1

switchport

vpc 1

lacp mode active

Long Distance

Dark Fiber

DC 1 DC 2C

OR

EA

GG

R

AC

CE

SS

Server Cluster

CO

RE

AG

GR

AC

CE

SS

Server Cluster

vPC domain 10 vPC domain 20

vPC domain 21vPC domain 11

vPC - Data Center Interconnect(DCI)

Rootguard

B

F

N

E

BPDUguard

BPDUfilter

Network port

Edge or portfast

- Normal port type

R

802.1AE (Optional)

E

- -

-

- -

E

E

E

E

F

F

F

F-

-

- -

-

--

B

N N

N

NN

N

R

R

-

RRRR

RR

NN

B

-

E

Design Best PracticesvPC as Data Center Interconnect (DCI)

PROS

• vPC is easy to configure and it provides robust and resilient interconnect solution

CONS

• Maximum of only two Data Centers can be interconnected

• Layer 3 peering between Data Centers cannot be done through vPC and separate links are required

Design Best PracticesvPC -Data Center Interconnect (DCI)

• vPC Domain id for vPC layers should be UNIQUE

• BPDU Filter on the edge devices to avoid BPDU propagation

• STP Edge Mode to provide fast Failover times

• No Loop must exist outside the vPC domain

• No L3 peering between Nexus 7000 devices (i.e. pure layer 2)

Dynamic routing over vPC ?

Dynamic routing over vPCUse Case 1 : Firewall at Aggregation layer

L3 Cloud

S1S2

FW-A FW-B

Dynamic Peering Relationship

• Peering Firewalls in routed mode over vPC

• Firewalls may be in active-standby mode

• Static routing / L3 P2P links NOT required

• External and internal traffic traverse same

port channel to firewall.

Dynamic routing over vPCUse Case 2 : Remote Orphan Site Peering in DCI Deployment

• vPC as Data Center Interconnect (DCI)

• Each Switch has routing adjacency withboth vPC device in other DC

• Each DC connected to a remote site byorphan port

• Remote sites forms routing adjacencywith both peers of its directly connectedDC

Remote Site 1 Remote Site 2

S1 S2

S3 S4

Dynamic Routing over vPCNew Supported Designs

Dynamic routing over vPCSupported Designs

P

PP

PP

PP

Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2.

Supported on Nexus 9X00 in ACI mode

Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card

Layer 3 services devices with vPC Layer 3 over DCI - vPC

Dynamic routing over vPCSupported Designs

P

PP

P

PP

STP inter-connection using a vPC VLAN Orphan device with vPC peers over vPC VLAN

Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2.

Supported on Nexus 9X00 in ACI mode

Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card

B

SVI 1

IP Y

VIP A

SVI 1

IP Z

VIP A

S1 S2

A

Router

Dynamic Routing over vPC

• Don’t attach routers to VPC domain via L2 port-channel

• Common workarounds:

• Individual L3 links for routed traffic

• Static route to FHRP VIP

Router

L3 ECMP

Router

SVI 1

IP Y

VIP A

SVI 1

IP Z

VIP A

SVI 1

IP Y

VIP A

SVI 1

IP Z

VIP A

SVI 2

IP X

SVI 2

IP X

SVI 2

IP X

Static Route to VIP A

S1 S1 S2S2

Devices without L3 over vPC support

Design Best PracticesvPC and Multicast

Source • vPC supports PIM-SM only

• vPC uses CFS to sync IGMP state

• Sources in vPC domain

− both vPC peers are forwarders

− Duplicates avoided via vPC loop-avoidance logic

• Sources in Layer 3 cloud

− Active forwarder elected on unicast metric

− vPC Primary elected active forwarder in case metric

are equal

Receivers

S2S1

Source

vPC : Get it Right the very First time

Agenda

• vPC Operations and Upgrade− vPC Self Isolation

− vPC Shutdown

− Graceful Insertion and Removal

− ISSU / ISSD with vPC

VPC Self-isolation Automatically triggered isolation

Example Presented: All Line Cards Fail

Current Impact Self-isolation Behavior (7.2)

• When this failure

happens on primary,

peer-link is brought down.

• This causes the

secondary brings down

all legs.

• Traffic is completely

blocked.

When this failure happens:

• Physically bring down peer-link

• Physically bring down all vPC legs

• Send self-isolation through peer-keep-

alive

Peer switch:

• Receive self-isolation from the peer

through peer-keep-alive

• Change role to Primary

• Bring up all down vPC legs

PKA

Vlan 1-100

Vlan 1-100Vlan 1-100

Primary Secondary

Current

PKA

Vlan 1-100

PrimarySecondary

Self-isolation

NOTE: Available in NX-OS 7.2, 5k/6k/7k

VPC Self-isolation Automatically triggered isolation

No up vlans on peer-link: This case address the issue that no vlansare up on the peer-link while the port channel is physically up (i.e., peer-link is up with no vlans). For example: vlan misconfiguration, hardware programming failure.

Current Behavior Self-isolation Behavior

• The up VLAN on vPC legs

are the results of configured

VLAN intersected with the

UP VLAN on peer-link (i.e.,

three-way intersection).

• All VLAN down on vPC

domain.

• This completely blocks the

traffic

Isolated switch:

• Physically bring down peer-link

• Physically bring down all vPC legs

• Send “self-isolation” through peer-keep-

alive

Peer switch:

• Receive self-isolation from the peer

through peer-keep-alive

• Change role to Primary

• Bring up all down vPC legs

PKA

No vlan

Vlan 1-100 Vlan 1-100

Current

PKA

Vlan 1-100

Self-

isolation

NOTE: Available in NX-OS 7.2, 5k/6k/7k

vPC Shutdown

vPC Configuration Best Practices

• Isolates a switch from the vPC complex

• Isolated switch can be debugged, reloaded, or

even removed physically, without affecting the

vPC traffic going through the non-isolated switch

Primary Secondary

vPC

Note : This Feature is currently not supported on Nexus 3X00 and 9X00 series

S1 S2

S3

switch# configure terminal

switch(config)# vpc domain 100

switch(config-vpc)# shutdown

vPC vPC

system mode maintenance

One command!Pre-change System Snapshot

Change window begins.

Graceful Insertion and Removal

vPC vPC

One command!Pre/Post-change Snapshot Comparison

Change window complete.

system mode normal

Graceful Insertion and Removal

Graceful Insertion and Removal

• Flexible framework providing a comprehensive, systemic method to isolate a node.

• Configuration profile foundation in NX-OS

• Initial support for:• vPC/vPC+• ISIS• OSPF• EIGRP• BGP• Interface

• Per VDC on Nexus 7x00

Platform Release

Nexus 5x00/6000 NX-OS 7.1

Nexus 7x00 NX-OS 7.2

Nexus 9000 NX-OS 7.X

ISSU / ISSD with vPC

• ISSU is the recommended system upgrade in a multi-device vPC environment

• vPC system can be independently upgraded with no disruption to traffic

• Upgrade is serialized and must be run one peer at a time (config lock will prevent synchronous upgrades)

• Configuration is locked on “other” vPC peer during ISSU

• Similar process of downgrades (ISSD)

• Check ISSU / ISSD compatibility matrix & ensure ISSU is supported from current to target release

5.2(x) / 6.2(x)

Agenda

• vPC with Fabric Technologies− vPC with Fabricpath (vPC+)

− vPC with FCOE

− vPC with VXLAN

− vPC with ACI

FabricPath: an Ethernet FabricShipping on Nexus 7x00, Nexus 600x and Nexus 5x00

N7K(config)# interface ethernet 1/1

N7K(config-if)# switchport mode fabricpath

• Eliminates Spanning tree limitations

• High resiliency, fast network re-convergence

• Any VLAN, Anywhere in the Fabric

• Connect a group of switches using an arbitrary topology

• With a simple CLI, aggregate them into a Fabric

FabricPath

• Physical architecture of vPC and vPC+ is the same from the access edge

• Functionality/Concepts of vPC and vPC+ are the same

• Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port

• vPC+ is not supported on Nexus 9X00 & Nexus 3X00 Series

Architecture of vPC and FabricPath with vPC+

VPC vs VPC+

FP VLAN’s

vPC+vPC

FP Port

FP

CE Port

CE VLAN’s

CE

Dynamic Routing over vPC+

P Routing Protocol Peer

N55xx, N56xx, N6000

Router/ Firewall

Dynamic Peering Relationship

• Layer 3 devices can form routing adjacencies with both the vPC+ peers over vPC

• The peer link ports and VLAN are configured in FabricPath mode.

• N55xx, N56xx, N6000 support this design with IPv4/IPv6 unicast and PIM-SM multicast

• This design is not supported on N7X00

Fabricpath Link

vPC

P P

P

Fabricpath Core

vPC with FCoEUnified Fabric Design

• vPC with FCoE is ONLY supported between hosts and N5K/N6K or N5K/N6K & N2232 pairs.

• Must follow specific rules:

• A ‘vfc’ interface can only be associated with a single-port port-channel.

• While the port-channel configurations are the same on both switches, the FCoE VLANs are different.

• FCoE VLANs are ‘not’ carried on the vPC peer-link (automatically pruned):

• FCoE and FIP ethertypes are ‘not’ forwarded over the vPC peer link.

• vPC carrying FCoE between two FCF’s is NOT supported.

• Best Practice: Use static port channel rather than LACP with vPC and boot from SAN.[If NX-OS is prior to 5.1(3)N1(1)]

VLAN 10,30

VLAN 10,20

STP Edge Trunk

VLAN 10 ONLY HERE!

Fabric A Fabric BLAN Fabric

Nexus 5000

FCF-A

Nexus 5000

FCF-B

vPC contains only 2 X 10GE

links – one to each Nexus 5X00

Problems being addressed:

• VLAN scale – VXLAN extends the L2 segment ID field to 24-bits, potentially allowing for up to 16 million unique L2 segments over the same network

• Layer 2 segment elasticity over Layer 3 boundary – VXLAN encapsulates L2 frame in IP-UDP header

High Level Technology Overview:

• MAC-in-UDP encapsulation.

• Leverages multicast in the transport network to simulate flooding behavior for broadcast, unknown unicast and multicast in the same segment

• Leverage ECMP to achieve optimal path usage over the transport network

Why VXLAN ?

VXLAN Packet Format

FCSOuter

Mac Header

Outer

IP HeaderUDP Header VXLAN

HeaderOriginal L2 Frame FCS

Dst.

MA

C A

ddr.

Src

.

MA

C A

ddr.

VLA

N T

ype

0x8100

VLA

N I

D

Ta

g

Eth

er

Type

0x0800

IP H

eader

Mis

cD

ata

Pro

tocol

0x11

Header

Checksum

Oute

r

Src

. IP

Oute

r

Dst.

IP

UD

P

Src

. P

ort

VX

LA

N P

ort

UD

P L

ength

Checksum

0x0000

VX

LA

N

RR

RR

1R

RR

Reserv

ed

VN

ID

Reserv

ed

14 Bytes

(4 bytes optional) 20 Bytes 8 Bytes 8 Bytes

48 48 16 16 16 72 8 16 32 32 16 16 16 16 8 24 24 8

• VXLAN is a Layer 2 overlay scheme over a Layer 3 network.

• VXLAN uses Ethernet in UDP encapsulation

• VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments

For YourReference

VXLAN TerminologyVTEP – Virtual Tunnel End Point

• VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).

• VTEP has two interfaces :

1. Bridging functionality for local hosts

2. IP identification in the core network for VXLAN encapsulation / de-encapsulation.

Local LAN Segment

IP Interface

End SystemEnd System

VTEP

Transport IP Network

Local LAN Segment

IP Interface

End SystemEnd System

VTEP

vPC VTEP

VXLAN

VLAN

vPC VTEP vPC VTEP

• When vPC is enabled an ‘anycast’ VTEP

address is programmed on both vPC

peers

• Multicast topology prevents BUM traffic

being sent to the same IP address across

the L3 network (prevents duplication of

flooded packets)

• vPC peer-gateway feature must be

enabled on both peers

• VXLAN header is ‘not’ carried on the vPC

Peer link

VXLAN & VPCVPC Configuration

vtep

1

vtep

2

vtep

3

vtep

4

H110.10.10.10

VLAN 10(vpc)

H210.10.10.20

VLAN 10(vpc)

VTEP1

vlan 10

vn-segment 10000

interface loopback 0

ip address <VTEP individual IP – orphan)

ip address <VTEP anycast IP – per VPC domain> secondary

!

interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

Map VNI to VLAN

VXLAN Tunnel Interface

Source Interfaceindividual IP is used for single attached Hostsanycast IP is used for VPC attached Hosts

VTEP2

vlan 10

vn-segment 10000

interface loopback 0

ip address <VTEP individual IP - orphan>

ip address <VTEP anycast IP – per VPC domain> secondary

!

interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

For YourReference

VXLAN & VPCVPC Configuration

vtep

1

vtep

2

vtep

3

vtep

4

H110.10.10.10

VLAN 10(vpc)

H210.10.10.20

VLAN 10(vpc)

VTEP1

vlan 10

vn-segment 10000

interface loopback 0

ip address 1.1.1.1/32

ip address 1.1.1.201/32 secondary

!

Interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

VTEP3

vlan 10

vn-segment 10000

interface loopback 0

ip address 1.1.1.3/32

ip address 1.1.1.202/32 secondary

!

Interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

VTEP2

vlan 10

vn-segment 10000

interface loopback 0

ip address 1.1.1.2/32

ip address 1.1.1.201/32 secondary

!

Interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

VTEP4

vlan 10

vn-segment 10000

interface loopback 0

ip address 1.1.1.4/32

ip address 1.1.1.202/32 secondary

!

Interface nve1

source-interface loopback0

member vni 10000 mcast-group 235.1.1.1

For YourReference

VXLAN & VPCDual attached Host to dual attached Host (Layer-2)

vtep

1

vtep

2

vtep

3

vtep

4

H110.10.10.10

VLAN 10(vpc)

H210.10.10.20

VLAN 10(vpc)

• Host 1 (H1) and Host 2 (H2) are dual

connected to a VPC domain

• As H1 is behind a VPC interface, the

anycast VTEP IP is the source for

the the VXLAN encapsulation

• As H2 is behind a VPC interface, the

anycast VTEP IP is the target

Nexus 9000 + APIC = ACI

APICAPIC

APIC

External

Network

App DBWeb

QoS

Filter

QoS

Service

QoS

Filter

ACI uses a policy based approach that focuses on the application.

vPC and ACIACI fabric utilised for control-plane vPC

Domains

vPC peers

ACI

fabric

• No dedicated peer-link between vPC peers:

Fabric itself serves as the MCT

vPC vPC• CFS (Cisco Fabric Services) is replaced by Zero Message Queue (ZMQ)

vtep

1

vtep

2

vtep

3

• No out-of-band mechanism to detect peer liveliness:

Due to rich fabric-connectivity (leaf-spine), it is very unlikely that peers will have no active path between them

• As ACI fabric is VXLAN-based, an anycast VTEP is shared by both leaf switches in a vPC domain

Agenda

• Scalability

vPC Scalability

For Latest Scalability numbers please refer to the scalability limits pages for the platform

Nexus 7X00

http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NX-OS_Verified_Scalability_Guide.html

Nexus 5X00http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus5600/sw/verified_scalability/701N11/b_N5600_Verified_Scalability_701N11/b_N6000_Verified_

Scalability_700N11_chapter_01.html

Nexus 600X http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus6000/sw/verified_scalability/602N21/b_N6000_Verified_Scalability_602N21/b_N6000_Verified_

Scalability_602N12_chapter_01.html

Nexus 9X00

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/7-x/scalability/guide_703I12/b_Cisco_Nexus_9000_Series_NX-

OS_Verified_Scalability_Guide_703I12.html

Nexus 3X00 http://www.cisco.com/en/US/docs/switches/datacenter/nexus3000/sw/configuration_limits/503_u5_1/b_Nexus3k_Verified_Scalability_503U51.html

84

Agenda

• Reference Material

Reference Material

• vPC Best Practices Design Guide: http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guid

e.pdf

• vPC design guides:

http://www.cisco.com/en/US/partner/products/ps9670/products_implementation_design_guides_list.html

• vPC and VSS Interoperability white Paper:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_589890.html

• VXLAN Overview :

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.html

• Fabrcipath whitepaper :

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-687554.html

ACI Overview

http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/aci-fabric-controller/white-paper-c11-729587.html

For YourReference

vPC in 2015

vPC Benefits

Fabricpath

VXLAN

Key Take-Aways

• No Blocked Ports

• High availability

• Fast Convergence

• Eliminates Spanning-Tree *

• High resiliency

• vPC+ for legacy switches,

servers, hosts

• L2 segment scalability

• VTEP redundancy with

vPC

VXLAN, ACI, Fabricpath

• Policy Based

• Fabric for vPC control

plane

ACI

• Unified Fabric for LAN &

SAN

FCoE

Participate in the “My Favorite Speaker” Contest

• Promote your favorite speaker through Twitter and you could win $200 of Cisco Press products (@CiscoPress)

• Send a tweet and include

• Your favorite speaker’s Twitter handle

• Two hashtags: #CLUS #MyFavoriteSpeaker

• You can submit an entry for more than one of your “favorite” speakers

• Don’t forget to follow @CiscoLive and @CiscoPress

• View the official rules at http://bit.ly/CLUSwin

Promote Your Favorite Speaker and You Could Be a Winner

Complete Your Online Session Evaluation

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online

• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.

• Complete your session surveys though the Cisco Live mobile app or your computer on Cisco Live Connect.

Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Table Topics

• Meet the Engineer 1:1 meetings

• Related sessions

Thank you