Upload
buidieu
View
230
Download
6
Embed Size (px)
Citation preview
Network Management Automation of Cisco Nexus Fabrics
Tom Nosella, Sr. Director, Technical Marketing
BRKDCT-2444
• Fabric Management Challenges
• The 4 Slide VXLAN Primer
• The Nexus Fabric Manager
• Building a Managed Fabric
• Connecting to the Fabric
• Expanding the Fabric
• Upgrading the Fabric
• Conclusion
Agenda
Fabric Management Challenges
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Datacenter Fabric Management Challenges
3. Rapid rollout of fabric infrastructure• Need to respond to needs of the business
• Both initial installation and fabric expansion
2. Minimize fabric downtime• Eliminate misconfigurations (high cause of downtime)
• Rapid recovery of fabric outages
1. Want to take advantage of new protocols and architectures• IT operations expertise requirements for fabric management proficiency
• New protocols and architectures come with complexity challenges
BRKDCT-2444 5
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Many Approaches to Fabric Management
• CLI – switch-by-switch – most common• Unnecessarily inefficient and highly error prone
• Requires extensive knowledge of protocols and syntax
• Scripting – to CLI and/or API• Achieves some efficiency – requires devops expertise
• Geared mostly to static config snippets and software management
• Off-the-shelf management solution• Largely element management focused – switch-by-switch
• Some limited templating capabilities – still require CLI/protocol expertise
BRKDCT-2444 6
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
How Do We Achieve High Management Efficiency?
AUTOMATIONWhat is the right model for you?
CLI ACIScriptingElement
Management
CLI InteractionAutonomous System
FABRIC MANAGEMENT AUTOMATION
???
BRKDCT-2444 7
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Desirable Traits of a Fabric Management System
Fabric awareness • Not just a switch is a switch – comprehension of topology and architecture
Workflow oriented• Closer alignment with application/business needs – less focus on CLI/protocols
Self managing, self configuring• System can build and maintain fabric configuration based on workflow outputs
Extendable (API) • Ability to tie system into higher level orchestration system
Full lifecycle management• Ongoing management services throughout all phases of fabric lifecycle
BRKDCT-2444 8
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creation Expansion
FaultsReporting
Connection
FABRICMANAGER
Fabric Management Lifecycle and Considerations
• New switch bring-up• Zero-touch experience
• Initial switch configuration
• Fabric layout discovery
• Infrastructure configuration
• Device discovery
• Single and dual-homed hosts
• Broadcast domains
• Gateway functions
• Adding switch (leaf or spine)• Zero-touch experience
• Dynamic configuration
• Cabling verification
• Broadcast domain expansion
• Fault management system
• Self-resolution
• External notifications
• Switch RMA process
• Task log – who, what, when
• Object-based history/logs
• Logical/physical performance
• Fabric inventory
BRKDCT-2444 9
The 4 Slide VXLAN Primer
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What is Virtual Extensible LAN (VXLAN)?
• VXLAN provides a network with segmentation, IP mobility, scale, and stability
• Standards based network overlay technology
• Layer-2 and layer-3 over standard routed network
• Leverages layer-3 ECMP – all links forwarding
• Increased name space to 16 million identifiers (24 bit)
• Segmentation and multitenancy
• Integration of physical and virtual endpoints
Layer-3Network
BRKDCT-2444 11
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VXLAN Important Terminology
• VXLAN Tunnel Endpoint (VTEP)
• Performs encapsulation/de-encapsulation
• Usually located in leaf layer
• Can be in hardware or software
• Virtual Network Identifier (VNI)
• Mapping of VLAN to VXLAN (eg VNI 5000 maps to VLAN 500)
• Multiple VLANs can map to same VNI
• Underlay Network
• The IP routed network upon which VXLAN is built
Layer-3Network
(Underlay)VN
I 5000
VLAN 500
VTEP
VTEP
BRKDCT-2444 12
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Important VXLAN Services
• VTEP Discovery
• Join multicast group for discovery
Ethernet VPN (EVPN) control plane - leverages BGP
• End Device (MAC) Discovery
• Flood-and-learn – requires multicast
Ethernet VPN (EVPN) control plane – leverages BGP
• Handling Broadcast, Unknown, Multicast (BUM)
• Multicast
Ingress replication – in hardware
Layer-3Network
(Underlay)VN
I 5000
VLAN 500
VTEP
VTEP
MAC : 11:11:11:11:11:11
MAC : 22:22:22:22:22:22
BRKDCT-2444 13
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VN
I 5000
Basic VXLAN Diagram
Layer-3 Network(Underlay)
VLAN 500
MAC1 : 11:11:11:11:11:11
MAC5 : 55:55:55:55:55:55
VTEP 1
MAC2 : 22:22:22:22:22:22 MAC3 : 33:33:33:33:33:33
VLAN 500 VLAN 400
VLAN 500
VLAN 400
MAC4 : 44:44:44:44:44:44
Interior GatewayProtocol (eg. OSPF)
MP-BGP/EVPN
MAC VNI NEXT HOP
MAC1 5000 VTEP1_IP
MAC2 5000 VTEP2_IP
MAC3 4000 VTEP2_IP
VTEP 2
VTEP 3
BRKDCT-2444 14
The Nexus Fabric Manager
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Cisco Nexus Fabric Manager
Intelligent Fabric Lifecycle Management
• Fabric-wide focus – auto-configuration and management of fabric
• Initial support for Cisco Nexus 9000 Familyrunning stand-alone NX-OS mode
• Automation based on knowledge of underlying fabric architecture
• Designed to simplify fabric management through its various lifecycle phases
• Delivered via VXLAN-based architectureFabric Management Lifecycle
Creation Expansion
FaultsReporting
Connection
FABRICMANAGER
BRKDCT-2444 16
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
What Does the Nexus Fabric Manager Do ?
Fabric Level Abstraction - Point-and-Click Interface
• Interaction via a simplified request model
• Say what you need, not how to do it
• Simplified point-and-click interface
• Focus on high ease of use
• Simplified tiles view for quick access andefficient management of numerous objects
• Intelligent live, actionable, topology mapping facility
Tiles View
Topology View
BRKDCT-2444 17
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric Management Workflows
Sample fabric management workflows
1. Create a fabric
• NFM creates and manages HA-enabled fabric
2. Add a new switch to the fabric
• NFM discovers, adds, and configures new switch
3. Create a broadcast domain
• NFM creates and manages VLANs and VXLAN topology
Assign VNID from NFM managed pool
Assign VLAN from NFM managed pool
Establish VLANport membership
Map VLAN to VNID on target leafs
Attach VNID to VTEP
• Optimized for fabric management workflows
• Help network ops quickly support business needs
• Switch features managed based on workflows
Add to broadcast domain
Build a
broadcast
domain
BRKDCT-2444 18
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus Fabric Manager Architectural Overview
• Physical appliance containing management engine and web UI
• Can manage N9K switch it has IP connectivity to (NX-OS mode)
• Communication with switches via NX-OS API
• Required initial switch configuration
1. Preconfigure mgmt IP, gateway, username/password and import switch
2. Leverage Auto Fabric Provisioning (AFP) via NFM embedded POAP services (zero touch)
Point-and-Click
User Interface
Fabric-Aware
Control Engine
RE
ST
AP
I
FABRICMANAGER
Mg
mtN
etw
ork
Sw
itch
Po
ol
BRKDCT-2444 19
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ma
na
ge
me
nt N
etw
ork
Managed and Monitored Objects
• Switches and interfaces in a switch pool can be in managed or monitored mode
• Imported switch initially set as monitored
• Only operational state and stats monitored
• Switch software upgrades supported
• API access and SNMP traps enabled
• Can set perimeter interfaces to monitored
• No changes performed by fabric manager
• Can perform custom configs via switch CLI• Eg. Custom ‘funky’ BGP config (not currently
supported by NFM) to uplink to a core network
MONITORED
MONITORED
MANAGED
MANAGED
MANAGED MANAGED
MANAGED
monitoredinterfaces
FABRICMANAGER
BRKDCT-2444 20
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Switch Interface Roles
Switchpool
?
Managed
Monitored
Foreign
Unknown – unknown device
Network Perimeter InterfacesConnected to outside switchpool (foreign)
Network Infrastructure InterfacesConnected to devices inside switchpool
Host-facing – connected to a host
Uplink – connected to router or L4/7
Managed
Monitore
d
Managed
Monitore
d
N/A Switch-facing – switch-to-switch linksN/A Peer-link – used for vPC
Switch-facing – connected to foreign switch
Known
Unknown
BRKDCT-2444 21
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Two UI Views – Both Searchable and Actionable
Tiles ViewTopology View
BRKDCT-2444 22
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Nexus Fabric Manager Tiles View
• Tiles are used to present information for numerous objects
• Efficient organization and quick retrieval of object details
Hyperlink to switch
interfaces tiles view
Hyperlink to candidate
vPC peer switch
Switch role
Switch model Multi-select
Switch
IP address
Switch name (CLI)
Switch name (NFM)
BRKDCT-2444 23
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
User Interface Object Search Capability
host11
Text searches
BRKDCT-2444 24
Building a Managed Fabric
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Starting Leaf-Spine Topology
• All switches are new or with erased configurations – ie. greenfield only
• NFM will not erase switches, so importing a partially configured switch could likely cause problems
BRKDCT-2444 26
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Let’s Describe Each Step
Architecture User Interface Command Line Interface
BRKDCT-2444 27
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Main User Interface Window
Multi-Edit
Contextual Menu
User Menu
Faults
Function Tabs
Filter/Sort Bar
Admin Menu
Main
Window
BRKDCT-2444 28
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Build a Fabric with the Nexus Fabric Manager
Rack the switches
Cable switches in leaf-spine topology
Boot switches with basic configuration or use Auto Fabric Provisioning
Discover switches via seed switch IP address within fabric manager
Select all switches and change to managed mode
Fabric is now managed
BRKDCT-2444 29
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Racking and Cabling the Switches
• Switch cabling must resemble two-tier leaf-spine architecture• NFM will verify topology and alert if required
• NFM shuts down invalid links
• Candidate vPC peer links between leafs• Will discover and designate as peer candidates
• No vPC configuration added until user instructs NFM to build host-facing vPC
• Hosts can be configured in single or multi-homed connection arrangement to leaf switches
Improper Cabling
Candidate
vPC Links
Ma
na
ge
me
nt N
etw
ork
FABRICMANAGER
BRKDCT-2444 30
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ma
na
ge
me
nt N
etw
ork
Booting the Switches• Switches require a few pieces of
basic configuration to be imported by NFM• IP_addr, IP_gateway, username/password
• Nexus 9500 series: L3 interfaces to be enabled for auto fabric discovery mode (via CDP)
FABRICMANAGER
APF
1. Basic configured switch - CLI console• Only require above – skip remainder
2. Auto Fabric Provisioning (AFP)• Enables the NFM to bootstrap new switches
• Based on Power-On-Auto-Provisioning (POAP)
• Import switches by their serial number
• Assign leaf/spine and configuration in one step
BRKDCT-2444 31
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
The Switchpool
• The switchpool is the container object within which all manageable fabric objects reside –highest level fabric object
• As switches are added to the switchpool, they can be managed or monitored by the NFM
• NFM currently supports one switchpool
• Foreign devices (hosts, switches) are considered always outside the switchpool
SWITCHPOOL
BRKDCT-2444 32
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Discovering and Importing a Fabric
• Four methods to import a fabric
• Basic configured switch – switch-by-switch – auto-discover turned off• Provide switch IP, import switch, discover neighbors, select neighbors and repeat –
switches in monitored mode
• Basic configured switch – auto-discover turned on• Provide seed switch IP, import switch, discover neighbors, and repeat until no supported
switches with same credentials found – switches in monitored mode
• Auto fabric provisioning (AFP) to monitored mode• Bootstrap switches and import as monitored mode switches
• Auto fabric provisioning (AFP) to managed mode• Bootstrap switches and import as managed mode switches
• Methods can be mixed to import fabric
BRKDCT-2444 33
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
BASIC Basic Configured Switch Import
IP address of switch
Serial number of switch
Must provide serial to activate
To access switch – can come from profile
To access switch – can come from profile
Switch name at CLI – optional
Local to NFM – optional
Auto / leaf / spine
Will override switchpool defaults
• This method assumes switches already have IP address and username/password configured
Can set desired image – no auto upgrade
Required
Optional
BRKDCT-2444 34
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Successful Basic Configured Switch Import
• Switches are in monitored mode by default• Unless imported via serial and set to managed
• Only configuration changes made to switches is viaSSH to switch to enable API access and set SNMP trap destination to NFMfeature nxapi
nxapi https port 443
nxapi use-vrf management
snmp-server host 172.31.160.88 traps version 2c agent_community udp-port 17015
snmp-server host 172.31.160.88 use-vrf management udp-port 17015
• NFM assigns unique SW# name to switchalong with actual switch name at CLI
• Must verify switch role to ensure discoveredrole is accurate
Monitored
Mode
(LEAF2)
(LEAF2)
BASIC
BRKDCT-2444 35
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Ma
na
ge
me
nt N
etw
ork
Fabric Discovered and in Monitored Mode
MONITORED
MONITORED
MONITORED
MONITORED
MONITORED MONITORED
Monitored
Mode
FABRICMANAGER
CDP/LLDP CDP/LLDP
BRKDCT-2444 36
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Auto Fabric Provisioning (AFP)
• Ability to pre-provision entire fabric and then simply turn switches on
• Pre-build switch objects in NFM using switch serial numbers
• Leverages NX-OS embedded POAP services
• Switches put into POAP mode by write erase and rebooting switch
• Switches continue POAP process until success or user interrupts
• Note: if0 must be configured with IP address and be reachable by booting switches and their DHCP requests – ie. same VLAN
• May see NFM fault as shown below if if0 not configured
BRKDCT-2444 37
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Auto Fabric Provisioning (AFP)
• Creates switch object and associated AFP profile to bootstrap and import switch
Desired address for new switch
Serial number of new switch
Managed/Monitored
To access switch – can come from profile
To access switch – can come from profile
New name to assign to switch
Role to assign to switch
Profile to assign to switch
Image to be upgraded to automatically
NFM-local description
Required
Optional
BRKDCT-2444 38
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Auto Fabric Provisioning (AFP) Process
• Can verify switch in POAP mode at switch console2016 Apr 17 14:55:14 switch %$ VDC-1 %$ %POAP-2-POAP_DHCP_DISCOVER_START: POAP DHCP Discover phase started
2016 Apr 17 14:55:22 switch %$ VDC-1 %$ %POAP-2-POAP_FAILURE: POAP DHCP discover phase failed
2016 Apr 17 14:55:29 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: USB Initializing Success
2016 Apr 17 14:55:29 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: USB disk not detected
2016 Apr 17 14:55:29 switch %$ VDC-1 %$ last message repeated 1 time
{repeats every 15 seconds}
• NFM will attempt to find the switch but will fail (this is normal)•
• Can verify AFP process is progressing by looking at switch console2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Using DHCP, information received over mgmt0 from 172.31.160.89
2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Assigned IP address: 172.31.160.34
2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Netmask: 255.255.255.128 Picked up from if0 config2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: DNS Server: 1.1.1.1 Assigned by default – 0K2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Default Gateway: 172.31.160.1 Picked up from if0 config2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Server: 172.31.160.88 Bootst2016 Apr 17 15:51:28 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: Script Name: poap.py
2016 Apr 17 15:51:41 switch %$ VDC-1 %$ %POAP-2-POAP_INFO: poap_dhcp_intf_ac_action_configuration_success: the script
download string is [copy tftp://172.31.160.88/poap.py bootflash:scripts/script.sh vrf management ]
. . .
2016 Apr 17 15:56:10 switch %$ VDC-1 %$ %POAP-2-POAP_SCRIPT_EXEC_SUCCESS: POAP script execution success
2016 Apr 17 15:56:11 switch %$ VDC-1 %$ %POAP-2-POAP_RELOAD_DEVICE: Reload device
Takes 5-7 minutes
BRKDCT-2444 39
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
• Moving switches to managed mode starts fabric configuration• Intra-fabric port channels (can be disabled)
• Underlay IP addressing
• Underlay interior gateway protocol (OSPF)
• Bi-directional Forward Detection (BFD)
• Multi-protocol BGP and route reflectors
• EVPN configuration
• VTEP and loopback interface creation
• Enable LLDP
Now For the Magic - Managed Mode
BRKDCT-2444 41
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Sample Applied Configuration in Spine Switches204 new lines of CLI added – per switch
Enabling Features1. feature nxapi2. nv overlay evpn3. feature ospf4. feature bgp5. feature interface-vlan6. feature vn-segment-vlan-based 7. feature lacp8. feature lldp9. feature bfd10. feature nv overlay
Enable EVPN
Enable VXLAN
Enable discovery
Enable VXLAN
Creating Port-Channels1. interface port-channel500 2. description This interface has been created
by Nexus Fabric Manager at 172.31.160.41. This port-channel was auto-created betweensw1 and sw2
3. no switchport4. mtu 92165. bfd interval 50 min_rx 50 multiplier 3 6. no ip redirects 7. ip address 10.0.0.15/31 8. no ipv6 redirects 9. ip ospf network point-to-point 10. ip router ospf 100 area 0.0.0.0 11. ip ospf bfd
Automatically added by NFM
Jumbo MTU
Auto addressing P2P links
Routing Protocols1. router ospf 100 2. bfd3. router-id 10.0.0.1 4. redistribute static route-map local-into-ospf5.
6. router bgp 655357. router-id 10.0.0.1 8. neighbor 10.0.0.3 remote-as 65535 9. remote-as 65535 10. update-source loopback501 11. address-family l2vpn evpn12. send-community both 13. route-reflector-client14. neighbor 10.0.0.4 remote-as 65535 15. remote-as 65535 16. update-source loopback501 17. address-family l2vpn evpn18. send-community both 19. route-reflector-client 20. neighbor 10.0.0.5 remote-as 65535 21. remote-as 65535 22. update-source loopback501 23. address-family l2vpn evpn24. send-community both 25. route-reflector-client
iBGP process - Private AS
Distribute EVPN info
Connect to route reflector
BRKDCT-2444 42
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Sample Applied Configuration in Leaf Switches215 new lines of CLI added – per switch
EVPN Configuration1. evpn2. vni 16777214 l2 3. rd auto 4. route-target import auto 5. route-target export auto
Special VNI for L3
VXLAN VTEP Interface1. interface nve1 2. no shutdown3. description Used by NFM for VXLAN termination4. source-interface loopback500 5. host-reachability protocol bgp6. member vni 16777214 associate-vrf
VXLAN VTEP interface
MP-BGP/EVPNSpecial VNI for L3
Loopbacks for VXLAN Underlay1. interface loopback500 2. description Used by NFM for VXLAN 3. termination (source-interface of nve1) 4. ip address 10.0.0.7/32 5. ip ospf network point-to-point 6. ip router ospf 100 area 0.0.0.0 7. ip ospf bfd8. interface loopback501 9. description Used by NFM for EVPN routing 10. ip address 10.0.0.4/32 11. ip ospf network point-to-point 12. ip router ospf 100 area 0.0.0.0 13. ip ospf bfd
Used for VTEP reachability via OSPF
Used for BGP reachability for EVPN
Underlay Routing Protocol1. router bgp 65535 2. router-id 10.0.0.4 3. neighbor 10.0.0.1 remote-as 65535 4. remote-as 65535 5. update-source loopback501 6. address-family l2vpn evpn7. send-community both 8. neighbor 10.0.0.2 remote-as 65535
9. remote-as 65535 10. update-source loopback501 11. address-family l2vpn evpn12. send-community both 13. vrf switchpool-default 14. address-family ipv4 unicast 15. advertise l2vpn evpn
One per spine
Default VRF for VXLAN routing
Underlay VRF1. vrf context underlay 2. address-family ipv4 unicast 3. vrf context switchpool-default 4. vni 16777214 5. rd auto 6. address-family ipv4 unicast 7. route-target both auto 8. route-target both auto evpn
Currently unused
Overlay routingfor switch pool
BRKDCT-2444 43
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Process Summary . . .
Managed Mode
MonitoredMode
BASIC
Managed Mode
BRKDCT-2444 44
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
User Interface Topology Clustered Leaf
Switches (vPC) Spine Switch Leaf Switch
Discovered Host
Host Interface
Switch Interface
Port Channel
BRKDCT-2444 45
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Switch and Interface Profiles
• Used to apply common feature configurations to groups of objects
• Can assign default profiles for certain objects - one per object• For all leaf, spine switches, for all host-facing, switch-facing, uplink interfaces
• Can be assigned to single switches, interfaces in object edit panel
• As new profile changes are made, CLI changes automatically pushed to switch(es)
BRKDCT-2444 46
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creating a Switch Profile
Default profiles dialog from switch pool settings panel
BRKDCT-2444 47
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
radius-server key 7 "ToIkLhPpG"
radius-server host 10.10.1.1 key 7 "ShMoMhTl" authentication accounting
aaa group server radius RadServer
server 10.10.1.1
Adding Extra CLI Configuration
• Can add CLI configuration snippets to switch profiles• Eg. Can be used to add a specific RADIUS/TACACS+ configuration
• No syntax validation, no automatic ‘no’ of commands if removed
• Object-specific profiles can be created with object-specific CLI snippets
BRKDCT-2444 48
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
MP-BGP
OSPF
Ma
na
ge
me
nt N
etw
ork
The Fabric is Now Managed
MANAGED
MANAGED
MANAGED
MANAGED
MANAGED MANAGED
VT
EP
VT
EP
VT
EP
VT
EP
FABRICMANAGER
CDP/LLDP CDP/LLDP
BRKDCT-2444 49
Connecting to the Fabric
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Steps to Connecting Devices to the Fabric
Perimeter Device Discovery
Port Channels and host-facing vPCs
Broadcast Domains
Gateways
Virtual Routing and Forwarding (VRF)
BRKDCT-2444 51
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Perimeter Device Discovery
• Foreign device - any device connected toperimeter interfaces on leaf switches
• Hosts, firewalls, other switches, etc
• To be discovered they must support and be advertising CDP and/or LLDP
• If not discovered, leaf switch interface must be manually assigned a role for the interface to be enabled – otherwise role remains undetermined and soft shutdown• Ie. active host with no agent will remain isolated
until role assigned to leaf switch perimeterinterface
• Neighbors tab shows all foreign devices attached to fabric
Foreign
Hypervisor
(vSwitch)
Foreign
host
Foreign
switch
BRKDCT-2444 52
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Foreign Device Discovery• Five categories of foreign devices - Host, hypervisor, networking, switch, unknown
• NFM processes 'Platform ID' from CDP, and/or 'System description' from LLDP
• NFM recognizes ESXi as 'hypervisor', KVM or generic Linux as 'host', and Nexus 5k, 7k, 9k as 'switch'. All other devices will be considered 'unknown’
• If device speaks CDP and LLDP, information from both is used
• Foreign device and foreign device interface objects are created
• Foreign device objects are persistent even if connected switches are deleted
Foreign hypervisor object Foreign interface object Foreign interface object
BRKDCT-2444 53
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Connecting a Server to the Fabric
• Hosts can be single or multi-homed
• Multi-homed servers may require Virtual Port Channel (vPC) to fabric
• Nexus Fabric Manager automatically identifies candidate vPC links as part of fabric discovery
• No configuration is pushed to switches until user action to build port channel or vPC
• Host-facing vPCs cannot use leaf-spine links for vPC peer (unlike ACI)
Switches Tab
Interfaces Tab
BRKDCT-2444 54
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creating a Port Channel (vPC)
filter
2
Filter and Select Host Interfaces
host13
1
filter
host13PC
BRKDCT-2444 55
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Port Channel (vPC) Notes• Each port channel has two types of IDs
• NFM-based unique ID referred to as logical ID• Switch-to-switch logical IDs start at 2000 – two per port channel
• Host-to-switch logical IDs start at 1 – one per port channel or vPC
• Switch-based ID referred to as physical ID• Actual ID as shown within switch CLI configuration
• Can be different on switches at either end
• Two per port channel (4 per vPC – two on each switch)
• Nexus fabric manager assigns all physical port channel IDs from 500 , vPC Domain IDs from 1
• User can manually add port channels at CLI with IDs below 500
Port Channels
L_ID2 = po2001L_ID1 = po2000
P_ID1 = po500 P_ID2 = po500
L_ID1 = po1
P_ID1 = po502P_ID2 = po503
P_ID1 = po503P_ID1 = po502
BRKDCT-2444 56
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Port Channel Example Logical ID (NFM)
Neighbor switchLogical ID (NFM)
Members (switch)
Physical ID (switch)
BRKDCT-2444 57
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Building a Broadcast Domain
• Connect devices at layer 2 – fabric wide
• Can assign VLAN ID or let NFM assign• NFM always assigns VNID
• One or more broadcast domains can be assigned to an interface• Interfaces always in VLAN trunking mode
• NFM automatically builds required VXLAN configuration on all switches
• Two methods• Select switch interfaces and assign – method 1
• Create broadcast domain then add interfaces – method 2
Ma
na
ge
me
nt N
etw
ork
MANAGED
MANAGED
MANAGED
MANAGED
MANAGED MANAGED
VT
EP
VT
EP
VT
EP
VT
EP
FABRICMANAGER
BRKDCT-2444 58
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
3
Interfaces selected to
build broadcast domain
Creating a Broadcast Domain
filter
2
Filter and Select Host Interfaces
host13
1
filter
BD_10
10 Optional
BRKDCT-2444 59
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Applied Configuration For New Broadcast Domain
Interface Configuration1. interface Ethernet1/12. switchport mode trunk3. switchport trunk allowed vlan 104. spanning-tree bpduguard enable
• Configuration built by Nexus Fabric Manager and pushed to leaf switches
• VNID assigned with fixed offset (20000 by default – configurable)
VXLAN VTEP Interface1. interface nve12. no shutdown3. description Used by NFM for VXLAN termination4. source-interface loopback5005. host-reachability protocol bgp6. member vni 200107. ingress-replication protocol bgp8. member vni 16777214 associate-vrf
Enabling ingress replication for BUM packets
VLAN/VNID Configuration1. vlan 1,10,3966-39672. vlan 103. vn-segment 200104. vlan 39675. vn-segment 16777214
New VLAN/VNI pair
EVPN Configuration1. evpn2. vni 20010 l23. rd auto4. route-target import auto5. route-target export auto6. vni 16777214 l27. rd auto8. route-target import auto9. route-target export auto
Enabling EVPN
BRKDCT-2444 60
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Broadcast Domain Notes
• Nexus Fabric Manager can automatically assign VLAN ID and will start at VLAN 2 • VXLAN VNID set as VLAN+offset (in settings – default 20,000)
• Broadcast domain only active with members, just like switch
• When interface is added to broadcast domain, it is put into VLAN trunking mode with VLAN enabled• Can set native untagged VLAN through interface settings
• Must still add to broadcast domain to enable native VLAN
BroadcastDomains
Just setting native VLAN in interface settings
1. interface Ethernet1/12. switchport mode trunk3. switchport trunk native vlan 24. switchport trunk allowed vlan none5. spanning-tree bpduguard enable
1. interface Ethernet1/12. switchport mode trunk3. switchport trunk native vlan 24. switchport trunk allowed vlan 25. spanning-tree bpduguard enable
Also adding it to the broadcast domain
BRKDCT-2444 61
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VT
EP
MANAGED
MANAGED
VT
EP
FABRICMANAGER
3 4
1 2
2Gateway IP: 20.20.20.1/24MAC: aa:aa:aa:aa:aa:aa
1Gateway IP: 10.10.10.1/24MAC: aa:aa:aa:aa:aa:aa
4Gateway IP: 20.20.20.1/24MAC: bb:bb:bb:bb:bb:bb
3Gateway IP: 10.10.10.1/24MAC: bb:bb:bb:bb:bb:bb
Building an IP (Anycast) Gateway
• The VXLAN architecture provides Anycast gateway function• Same IP gateway per broadcast domain on
each switch
• Common MAC address per switch for all broadcast domains - configurable
• Eliminates tromboning of traffic to reach gateway
• Routing can occurs between broadcast domain gateways on each switch
BRKDCT-2444 62
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Interfaces selected to
build broadcast
domain
Creating an IP Anycast Gateway
10.10.10.1/24
BD_10
10 Optional
• By creating a gateway as part of broadcast domain creation, a VXLAN anycastgateway is also created• Can go back and edit
broadcast domain to add gateway
• Can add gateway to different overlay VRF
BRKDCT-2444 63
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Applied Configuration For New Anycast Gateway
Broadcast Domain Switched Virtual Interface1. interface Vlan102. no shutdown3. vrf member switchpool-default4. bfd interval 50 min_rx 50 multiplier 35. no ip redirects6. ip address 10.10.10.1/247. no ipv6 redirects8. fabric forwarding mode anycast-gateway
• Anycast gateway MAC configurable – same for all broadcast domains
• Gateway automatically put into default overlay VRF (called underlay-l3)
• Can be added to new VRF in one step – covered in next section
Anycast Gateway MAC Address1. fabric forwarding anycast-gateway-mac CABB.D324.7D50
Configurable in settings
BRKDCT-2444 64
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Anycast Gateway Notes
• Because VLANs automatically added to each leaf switch in fabric, so is the gateway function
• DHCP relay configuration can be added at CLI level on gateway interfaces
• ARP suppression not currently automatically enabled – will be enabled in future release
AnycastGateways
BRKDCT-2444 65
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Building a VRF
• Virtual Routing and Forwarding (VRF) domain can be added to the fabric through UI
• Gateway objects can be added to VRFs
• Interfaces in L3 mode can be added to VRFs
• VRFs are added to all leaf switches
• Once anycast gateway added to VRF . . .
• Extra VNI and VLAN assigned to VRF (seen in CLI)• VNI used for layer 3 routing between networks
• VLAN not used (NX-OS requirement)
VLAN 3
VLAN 2
VLAN 3
VLAN 2
VLAN 2
VLAN 3
VLAN 3900 / VNI 23900
VLAN 3901 / VNI 23901
Gateways
Gateways Gateways
VRF 1
VRF 2
BRKDCT-2444 66
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Default [Switch_Pool] VRF
• Default VRF is called switchpool-default in CLI (Actually an overlay VRF)
• VLAN 3967 / VNI 16777214 are reserved should user create an gateway in the default VRF
• Not used otherwise
Default [Switch_Pool] Overlay VRF SVI Configuration1. interface Vlan39672. no shutdown3. vrf member switchpool-default4. no ip redirects5. ip forward6. ipv6 address use-link-local-only7. no ipv6 redirects
Default [Switch_Pool] Overlay VLAN/VNI1. vlan 39672. vn-segment 16777214
Assigned by NFMAssigned by NFM
Default [Switch_Pool] Overlay VRF1. vrf context switchpool-default2. vni 167772143. rd auto4. address-family ipv4 unicast5. route-target both auto6. route-target both auto evpn
Technically the ‘overlay’ VRF
Default [Switch_Pool] Overlay VRF BGP Configuration1. router bgp 655352. . . .3. . . .4. vrf switchpool-default5. address-family ipv4 unicast6. advertise l2vpn evpn
Enable EVPN advertisements
BRKDCT-2444 67
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creating a VRF
New VRF Configuration1. vrf context Tenant_22. address-family ipv4 unicast
• When VRF added and has no members, only one line of configuration added
Tenant_2
BRKDCT-2444 68
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Add Broadcast Domain Gateway to VRF1
Tenant_2
BRKDCT-2444 69
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Applied Configuration for VRF + Gateway
New VLAN/VNID Mapping1. vlan 22. vn-segment 200023. vlan 39654. vn-segment 167772135. vlan 39676. vn-segment 16777214
Added for L3 routing in VRF
User broadcast domain
New VRF Configuration1. vrf context Tenant_22. vni 167772133. rd auto4. address-family ipv4 unicast5. route-target both auto6. route-target both auto evpn
Added for L3 routing in VRF
New VRF SVIs1. interface Vlan22. no shutdown3. vrf member Tenant_24. bfd interval 50 min_rx 50 multiplier 35. no ip redirects6. ip address 10.10.10.1/247. no ipv6 redirects8. fabric forwarding mode anycast-gateway
9. interface Vlan396510. no shutdown11. vrf member Tenant_212. ip forward13. ipv6 address use-link-local-only
Added for L3 routing in VRF
New VRF BGP configuration1. router bgp 655352. router-id 10.0.0.53. . . .4. . . .5. vrf Tenant_26. address-family ipv4 unicast7. advertise l2vpn evpn
New VRF Configuration1. interface nve12. no shutdown3. description Used by NFM for VXLAN termination4. source-interface loopback5005. host-reachability protocol bgp6. member vni 16777213 associate-vrf7. member vni 200028. ingress-replication protocol bgp9. member vni 16777214 associate-vrf
BRKDCT-2444 70
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
VRF Notes
• When adding broadcast domains, must be aware of VLANs assigned by system to accommodate anycast gateways in non-default VRFs
• No overlay routing protocol enabled in VRF• All anycast gateways in VRF will route locally between them
• Can manually add network advertisement statements to BGP/EVPN configuration in specific switch for given VRF
VRFs
1. router bgp 655352. . . .3. . . .4. vrf Tenant_25. address-family ipv4 unicast6. network 100.100.100.0/247. advertise l2vpn evpn
Advertise network to other leafs
BRKDCT-2444 71
Expanding the Fabric
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Expanding the Fabric• Step 1 – rack and cable new
switches in leaf-spine topology
• Step 2a – power on and provide switches with IP addr, gateway, username/passwd
• Switches will pop up in UI as foreign and can be imported from there
• Step 2b – preconfigure new switch objects using auto fabric provisioning(AFP) and then simply turn on switches
• NFM will bootstrap switches via POAP, import them, and build their entire configuration
MP-BGP
OSPF
Ma
na
ge
me
nt N
etw
ork
MANAGED
MANAGED
VT
EP
VT
EP
VT
EP
FABRICMANAGER
CDP/LLDP CDP/LLDP
MANAGED
MANAGED
VT
EP
CDP/LLDP
MANAGED MANAGED
BRKDCT-2444 73
Upgrading the Fabric
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Upgrading the Fabric
• Switches are added into ‘upgrade groups’• Image and upgrade policy are applied to groups
• Images are stored in Nexus Fabric Manager
• Edit groups to change image to upgrade to next release
BRKDCT-2444 75
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Creating Upgrade Group
Backup_All
7.0(3)I2(2a) (nxos.7.0.3.I2.2a.bin)
BRKDCT-2444 76
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Running an Upgrade Task
• Two backup strategies• Parallel and sequential
• Create ‘salt-and-pepper’ or ‘left-and-right’ upgrade groups
• Reuse group with new releases
BRKDCT-2444 77
Conclusion
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Conclusion
• Fabric awareness in a management platform leads to automation
• The less reliance on CLI and protocol knowledge – the faster to results
• The new Nexus Fabric Manager delivers automation and simplification of fabric lifecycle management
• The north-bound API of the Nexus Fabric Manager lends itself to integration into higher level orchestration
• The Nexus Fabric Manager live demo is in Cisco Datacenter display
BRKDCT-2444 79
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Complete Your Online Session Evaluation
Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at CiscoLive.com/Online
• Give us your feedback to be entered into a Daily Survey Drawing. A daily winner will receive a $750 Amazon gift card.
• Complete your session surveys through the Cisco Live mobile app or from the Session Catalog on CiscoLive.com/us.
BRKDCT-2444 80
© 2016 Cisco and/or its affiliates. All rights reserved. Cisco Public
Continue Your Education
• Demos in the Cisco campus
• Walk-in Self-Paced Labs
• Lunch & Learn
• Meet the Engineer 1:1 meetings
• Related sessions
BRKDCT-2444 81
Thank you