Cisco Application Virtual Switch Solution Guide - Cisco - Global Home Page€¦ · 2.1 AVS for Application Centric Infrastructure The Cisco AVS is integrated with the Cisco Application

© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information.

Cisco AVS Solution Guide

Cisco Application Virtual Switch Solution Guide Sep 3rd, 2014

© 2014 Cisco and/or its affiliates. All rights reserved. This document is Cisco Confidential Information. Page 2


1 Purpose ............................................................................................................................................................ 4

1.1 Pre-Requisite ....................................................................................................................................... 4

2 AVS Introduction .......................................................................................................................................... 5

2.1 AVS for Application Centric Infrastructure ............................................................................... 5

2.2 Cisco ACI Fabric Overview ............................................................................................................... 5

2.2.1 End Point Groups (EPGs) Concept ......................................................................................................... 6

2.2.2 OpFlex Protocol .............................................................................................................................................. 6

3 AVS Switching Modes .................................................................................................................................. 7

3.1.1 No Local Switching Mode ........................................................................................................................... 7

3.1.2 Local Switching .............................................................................................................................................. 8

4 Switch Failover and Link Aggregation .................................................................................................. 9

4.1 Port-Channel Technology ................................................................................................................ 9

4.1.1 LACP .................................................................................................................................................................... 9

4.1.2 Standard Port Channel ................................................................................................................................ 9

4.1.3 Virtual Port Channel ..................................................................................................................................... 9

4.1.4 Static Port-Channel ....................................................................................................................................... 9

4.2 MAC Pinning ...................................................................................................................................... 10

4.3 Virtual Port Channel (vPC) ........................................................................................................... 11

5 AVS Recommended Topologies ........................................................................................................... 12

5.1 Topology #1 AVS Host Directly Connected to Leaf .............................................................. 14

5.2 Topology#2 AVS Host Connected to Leaf via FEX ................................................................. 15

5.3 Topology#3 AVS Host Connected to Leaf via UCS FI ............................................................ 17

5.4 Topology#4 AVS Host Connected to Leaf via Switch ........................................................... 18

5.4.1 Double-Sided VPC with Nexus 5000 and AVS with MAC Pinning ........................................... 18

5.4.2 Double-Sided VPC with Nexus 5000 and AVS with VPC ............................................................. 19

5.5 Topology#5 AVS Host Connected to Leaf via Switch-FEX .................................................. 20



5.6 Topology#6 AVS Host Connected to Leaf via Multiple Switches ..................................... 21

5.7 Topology#7 AVS Host Connected to Leaf via UCS FI and Switch ..................................... 22

5.7.1 Single-Side VPC with Nexus 5000/UCS FI and AVS with MAC Pinning ................................ 22

6 AVS Implementation Best Practices ................................................................................................... 23

7 FAQ ................................................................................................................................................................. 24

7.1 Support Table ................................................................................................................................... 24

8 References ................................................................................................................................................... 25



1 Purpose

This document is intended as a solution-level reference for technical professionals responsible for preparing, planning, and implementing the Cisco Application Virtual Switch (AVS) for Data Center customer.

This document provides AVS planning considerations and topology recommendations, but does not discuss all the foundational technologies, procedures and best practices for deploying the routing, switching and data center setup required by the solution. Instead, it refers to detailed documents that discuss those technologies and implementation methods, while focusing on specific configuration and topologies for deploying AVS within the ACI (Application Centric Infrastructure) Solution.

1.1 Pre-Requisite

This document assumes that readers have thorough understanding of the Cisco ACI (Application Centric Infrastructure) and other Cisco Data Center technologies. Please refer to following links to understand these concepts.

http://www.cisco.com/go/aci

http://www.cisco.com/go/datacenter

http://www.cisco.com/go/aci

http://www.cisco.com/go/datacenter



2 AVS Introduction

Cisco Application Virtual Switch (AVS) is a hypervisor-resident distributed virtual switch that is specifically designed for the Cisco Application Centric Infrastructure (ACI) and managed by Cisco APIC (Application Policy Infrastructure Controller).

2.1 AVS for Application Centric Infrastructure The Cisco AVS is integrated with the Cisco Application Centric Infrastructure (ACI). Unlike Nexus1000V where management is done by a dedicated Virtual Supervisor Module, Cisco AVS is managed by the Cisco APIC. Cisco AVS implements the OpFlex protocol for control plane communication.

Before we dive into the specifics of AVS, it is important to understand the basic concepts about Cisco Application Centric Infrastructure fabric.

Note:

Cisco AVS is the Cisco vSwitch for ACI mode. If you are running Nexus 9000 in standalone mode, then you can use Nexus 1000V as vSwitch but otherwise you will use Cisco AVS.

2.2 Cisco ACI Fabric Overview The Cisco Application Centric Infrastructure Fabric (ACI) fabric includes Cisco Nexus 9000 Series switches with the APIC (Application Policy Infrastructure Controller) to run in the leaf/spine ACI fabric mode. In a recommended minimum configuration

• Three Cisco Nexus 9K (9500 series or 9336PQ) switches deployed as spines. • Only Cisco Nexus 9K switches (9300 Series) can connect to the spine switches as leaf

switches or nodes (All other devices, appliances and switches connect to the leaf nodes) • The APIC is an appliance running on Cisco UCS server and connects to one of the leaf

nodes. It manages the ACI fabric. The recommended minimum configuration for the APIC is a cluster of three replicated hosts. One can connect APIC to two different leaf switches for redundancy

The APIC fabric management functions do not operate in the data path of the fabric. The management is done via out of the band management network.

The following figure illustrates an overview of the leaf/spine ACI fabric.



The ACI fabric provides low-latency forwarding across high-bandwidth links (40 Gbps, with a 100-Gbps future capability). Traffic with the source and destination on the same leaf switch/node is handled locally. All other traffic traveling from the ingress leaf to the egress leaf goes through a spine switch. Although this architecture appears as two hops from a physical perspective, it is actually a single Layer 3 hop because the fabric operates as a single Layer 3 switch.

2.2.1 End Point Groups (EPGs) Concept

The central concept is to group endpoints (EPs) with identical semantics into endpoint groups (EPGs) and then write policies that regulate how such groups can interact with each other.

2.2.2 OpFlex Protocol

OpFlex, the southbound API, is an open and extensible policy protocol used to transfer abstract policy in XML or JavaScript Object Notation (JSON) between Cisco APIC and AVS switch.

http://tools.ietf.org/html/draft-smith-opflex-00

http://tools.ietf.org/html/draft-smith-opflex-00



3 AVS Switching Modes

Cisco AVS supports two modes of traffic forwarding:

1- No Local Switching Mode 2- Local Switch Mode

The forwarding mode is selected during Cisco AVS installation when the VMware vCenter Domain is created. VMware vCenter Domain creation is the step where APIC will communicate with vCenter and will dynamically create Cisco AVS. Following picture shows the different options on APIC controller

3.1.1 No Local Switching Mode “No Local Switching” mode was formerly known as FEX enable mode. In “No Local Switching” mode, all traffic (intra-EPG and/or inter-EPG) is forwarded by the physical leaf. In this mode, VXLAN is the only allowed encapsulation type.



3.1.2 Local Switching

“Local Switching” was formerly known as FEX Disable mode. In this mode all intra-EPG traffic is locally forwarded by the Cisco AVS, without the involvement of the physical leaf, if the traffic bound for the same host. All inter-EPG traffic is forwarded via the physical leaf.

In this mode, the Cisco AVS can use either use VLAN or VXLAN encapsulation for forwarding traffic to the leaf and back. The encapsulation type is selected during Cisco AVS installation.

• If VLAN encapsulation mode is used, a range of VLANs must be available for use by the Cisco AVS. These VLANs have local scope in that they have significance only within the Layer 2 network between the Cisco AVS and the leaf.

• If VXLAN encapsulation mode is used, only the infra-VLAN needs to be available between the Cisco AVS and the VXLAN. This results in a simplified configuration and is the recommended encapsulation mode if there are one or more switches between the Cisco AVS and the leaf.



4 Switch Failover and Link Aggregation

Network architects can use different approaches for protection against switch or link failover and link aggregation. The most common design approaches with Cisco AVS are virtual PortChannel (vPC) and MAC pinning. Both design approaches provide protection against single-link and physical-switch failures, but they differ in the way that the virtual and physical switches are coupled and the way that the VMware ESX or ESXi server traffic is distributed over the 10 Gigabit Ethernet links. The essence of all these approaches is Port-Channel technology.

4.1 Port-Channel Technology A Port-Channel (also referred to as Ether-Channel) on the Cisco AVS implements the standards-based IEEE 802.3ad or 802.1AX link aggregation protocol that incorporates the Link Aggregation Control Protocol (LACP) for automatic negotiation.

4.1.1 LACP

LACP dynamically bundle several physical ports together to form a single port channel. LACP enables a node to negotiate an automatic bundling of links by sending LACP packets to the peer node. LACP is simply a way to dynamically build Port-Channel. Essentially, the “active” end of the LACP group sends out special frames advertising the ability and desire to form a Port-Channel.

4.1.2 Standard Port Channel

Standard Port-Channel requires that all uplinks from one ESXi host in the Port-Channel group must be connected to single and same upstream physical switch.

4.1.3 Virtual Port Channel

When ESXi host uplinks are spread across more than one upstream physical switch, the upstream switches are clustered using Virtual Port-Channel (vPC).

4.1.4 Static Port-Channel

When LACP protocol is not running on the links, it is called Static Port-Channel mode. In Cisco OS and Cisco Nexus OS, Static Port-Channel mode is also called “Mode ON”. Following command displays the Static Port-Channel configuration on Nexus 5000 switch.

interface ethernet 1/1-2 channel-group 1 mode on



On the contrary, in APIC controller, LACP mode “Off” represents Static Port-Channel configuration.

A maximum of 16 links can be configured to form a Port-Channel group.

You can configure one of several types of port channel policies on the Cisco AVS: Link Aggregation Control Policy (LACP) in active or passive mode, MAC pinning, or static. You can configure port channel policies through the Cisco APIC GUI, the REST API, or the CLI.

MAC Pinning—MAC Pinning

Active—LACP active

Passive—LACP passive

Off— LACP Off (i.e Static Port-Channel)

4.2 MAC Pinning In a MAC Pinning mode, the Gigabit Ethernet uplinks from the Cisco AVS are treated as stand-alone links. In a two Gigabit Ethernet uplinks scenario, each Gigabit Ethernet interface is connected to a separate physical switch with Layer 2 continuity on all IEEE 802.1Q trunked VLANs between the two switches. Virtual Ethernet ports supporting virtual machines and vmkernel ports are allocated in a round-robin fashion over the available Gigabit Ethernet uplinks. Each MAC address is pinned to one of the uplinks until a failover event occurs. MAC pinning does not rely on any protocol to distinguish the different upstream switches, making the deployment independent of any hardware or design. This independence



enables consistent and easy deployment of the Cisco AVS, and it is the preferred method for deploying the Cisco AVS when the upstream switches cannot be clustered using Cisco vPC.

4.3 Virtual Port Channel (vPC) vPC is required on the upstream physical switches to enable the Port-Channel to span both upstream physical switches and still maintain availability for the VMware ESXi host should one switch fail or lose connectivity. This vPC clustering is transparent to the Cisco AVS. From AVS Host point of view, it sees the vPC cluster as one single switch.

For vPC to work, the Cisco Application Virtual Switch should be configured with LACP Port-Channel (configuration is done via APIC) with the two Gigabit Ethernet uplinks defined by one port profile.

The two upstream physical switches should be configured with vPC. The upstream switch (for example a Cisco Nexus 5000 or 7000 series switch) will appear as a single logical switch distributed over two physical chassis.

Differences Between vPC and MAC Pinning

Design Uplinks Physical-Switch Requirements

vPC Single logical Port-Channel

Clustered physical switches using a multichassis Ether-Channel (MEC) implementation such as Cisco vPC, virtual switching system (VSS), or virtual blade switch (VBS) technologies

MAC Pinning

All teamed uplinks in same Layer 2 domain

No special configuration other than Layer 2 continuity between both switches on all VLANs trunked to the VMware ESX or ESXi server

vPC is the recommended approach when vPC or clustered physical switches are available at the physical access layer. MAC pinning should be chosen when these options are not available.



5 AVS Recommended Topologies

On a very high level there are seven commonly deployed topologies supported by AVS. For ease of understanding these are divided into two groups

1- Standard Topologies • Topology#1 AVS host directly connected to N9K leaf switch • Topology#2 AVS host connected to N9K leaf switch via FEX • Topology#3 AVS host connected to N9K leaf switch via UCS FI

2- Extended Topologies • Topology#4 AVS host connected to N9K leaf via a single physical switch

i. Double-Sided VPC with Nexus 5000 and AVS with MAC Pinning ii. Double-Sided VPC with Nexus 5000 and AVS with VPC

• Topology#5 AVS host connected to N9K leaf via a switch-FEX • Topology#6 AVS host connected to N9K leaf via multiple switches • Topology#7 AVS host connected to N9K leaf via UCS FI and a Switch

i. Single-Side VPC with Nexus 5000/UCS FI and AVS with MAC Pinning

Leaf Switch

1

Leaf Switch

FEX

2

Leaf Switch

UCS FI

3



Leaf Switch Leaf Switch Leaf Switch Leaf Switch

Nexus 5K/6K/7K

5 6 7 4

Nexus 5K/6K/7K Nexus 5K/6K/7K Nexus 5K/6K/7k

UCS FI Nexus 5K/6K/7K Nexus 2000



5.1 Topology #1 AVS Host Directly Connected to Leaf In this topology ESXi host is directly connected to ACI Leaf Switch. This is typical scenario where a rack mount server (For example a Cisco UCS C-Series Server) is running ESXi hypervisor and AVS is running as a distributed virtual switch on it. It is recommended to use VLAN in this topology.



5.2 Topology#2 AVS Host Connected to Leaf via FEX In this topology ESXi host (For example a rack mount server like Cisco UCS C-Series Server) is connected to the FEX. FEX is then directly connected to APIC Leaf switch. VLAN local switching mode, VXLAN local switching mode, and VxLAN non-switching mode are supported with this topology. It is recommended to use VLAN with this topology.

There are some limitations with this topology that you should be aware of

• vPC is not supported between FEX and Leaf or for hosts directly connected to FEX • Only a single physical link is supported between the FEX and an ESX host connected to that FEX.

This means that LACP or MAC Pinning is not supported for ESXi hosts connected directly to an FEX that is connected directly to a leaf.



• For the single host connected between FEX and an ESXI host, when you choose an LACP mode

for a Cisco AVS, you should choose either MAC Pinning or Off on the APIC controller as shown in the following diagram

• These limitations does not apply to an extender that is connected to a Nexus 5000 or 7000 switch that is connected to a leaf (as shown in Topology#5)



5.3 Topology#3 AVS Host Connected to Leaf via UCS FI In topology#3, ESXi host is running on a Cisco UCS B-Series blade server. The B-Series server or the chassis is connected to UCS Fabric Interconnect. FI is then directly connected to ACI Leaf switch. This topology connects the ESX hypervisor to the Cisco APIC using via fabric interconnect, VPCs, LACP, and MAC pinning. In topology#3 one can use either VLAN or VXLAN. The main concept is to use VXLAN when there is more than one hop between ESXi host and Leaf Switch. Following picture shows the logical diagram of topology#3.



5.4 Topology#4 AVS Host Connected to Leaf via Switch This topology is a very common use case in scenario where for example customer has already deployed Cisco Nexus 5000, 6000 or 7000 and they want to have Cisco ACI fabric inserted in their current architecture. This topology connects the ESXi hypervisor to a Cisco APIC through the Cisco Nexus 5000 switch, virtual port channels, and MAC pinning. VXLAN will be used in this topology.

5.4.1 Double-Sided VPC with Nexus 5000 and AVS with MAC Pinning



5.4.2 Double-Sided VPC with Nexus 5000 and AVS with VPC



5.5 Topology#5 AVS Host Connected to Leaf via Switch-FEX This topology is not very different than topology#4. The only difference is that the L2 switch in the middle of AVS and Leaf switch has a Cisco Nexus 2000 Fabric Extender (FEX). And the ESXi host or AVS is connected to FEX. This is the most common scenario where FEX is deployed as a Top of the Rack (TOR) switch. VXLAN will be used in this topology. Topology#4 and 5 works almost the same in terms of multicast.



5.6 Topology#6 AVS Host Connected to Leaf via Multiple Switches This topology represents a customer running data center with core and aggregation architecture with Cisco Nexus 5000 or 7000 series switches. The customer wants to migrate to ACI based architecture in phases. The leaf switch could be connected to Nexus 5000 or 7000 switch at the aggregation layer. VXLAN is recommended in this topology.



5.7 Topology#7 AVS Host Connected to Leaf via UCS FI and Switch It is highly recommended to use VXLAN here because there are more than one hop between AVS and Leaf switch.

5.7.1 Single-Side VPC with Nexus 5000/UCS FI and AVS with MAC Pinning

This topology connects the AVS ESXi host to the leaf switches using MAC pinning, directly or via Cisco Nexus 5000 switches and Cisco UCS 62xx Series Fabric Interconnects.



6 AVS Implementation Best Practices

Following configuration are recommended when deploying Cisco AVS with the Cisco APIC

• Infra VLAN must be configured in the layer 2 network to establish connection between N9K leaf and AVS for OpeFlex.

• Infra VLAN must be configured on the leaf side port and AVS side ports of Cisco Fabric Interconnect

• DHCP relay policy must be configured for AVS so that the APIC can assign the IP address for the AVS vtep vmk in ESXi host

• Cisco Fabric Interconnect doesn’t support LACP on their southbound ports so it is recommended not to configure AVS with the LACP policy

• Configure vMotion on a separate VMKernel NIC with a dedicated EPG. Do not configure vMotion on the VMKernel NIC created for OpFlex channel

• One must not delete or change any parameters for the VMkernel NIC created for the OpFlex channel

• If VMkernel NIC created for the OpFlex channel is deleted by mistake, recreate it with the attach port-group vtep, and configure it with a dynamic IP address. One should never configure a static IP address for an OpFlex vmk NIC.



7 FAQ

7.1 Support Table

UCS port channel configuration is statically set to Link Aggregation Control Protocol (LACP) mode active. This configuration cannot be modified; therefore, all upstream port-channel configurations must adhere to LACP mode active as well. Alternatively, you can configure the upstream switch ports for LACP mode passive.

Following table list different switches and their supported port-channeling modes

LACP MAC Pinning Static Port Channel

VPC

AVS Yes Yes Yes Not Applicable

Nexus 9000 Leaf Yes Yes Yes Yes

UCS Fabric Interconnect

Yes (Northbound Ports)

No (Southbound Ports)

Yes No No

Nexus 5000 Nexus 7000

Yes Yes Yes Yes



8 References

Please refer to following supporting documents for more detailed information.

Cisco Application Virtual Switch

http://www.cisco.com/c/en/us/products/switches/application-virtual-switch/index.html

Cisco Application Centric Infrastructure Fundamentals http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-Fundamentals.html

Printed in USA 09/2014

http://www.cisco.com/c/en/us/products/switches/application-virtual-switch/index.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-Fundamentals.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-Fundamentals.html

Documents

Cisco Application Virtual Switch Solution Guide - Cisco - Global Home Page€¦ · 2.1 AVS for Application Centric Infrastructure The Cisco AVS is integrated with the Cisco Application