cumulusnetworks.com

Ravi Nittur - Dell EMCDiane Patton - Cumulus Networks

Building a Scalable Network for Private Cloud with Layer 3 and BGP

cumulusnetworks.com

A Layer 2 Data Center Environment

cumulusnetworks.com

Why a Layer 3 Clos Network?

Scalable network topology

Reliance on ECMP leads to simple IP-based fabrics

Fine grained failure domain

Predictable latency

Coupled with network virtualization, serves as a basis for agility and flexibility

SPINE

LEAF

cumulusnetworks.com

Which Routing Protocol for Clos?

eBGP in a Data Center● Simple● Scalable

○ Powers the Internet● Multiprotocol● Traffic Engineering● Filtering capabilities

spine

leaf

ECMP ECMP ECMP

Layer 3

RFC 7938 provides more Information on large DC

cumulusnetworks.com

2 Tier Clos with eBGP

leaf

AS65100

AS65200 AS65201 AS65203 AS65204AS65202

spine

Layer 3

cumulusnetworks.com

Can grow… 3 Tier Clos

AS65000

AS65100

AS65200 AS65210POD A

AS65200 AS65201

AS65101

POD B

cumulusnetworks.com

Layer 3 Routing

Cumulus Routing on the Host

spine

leaf

ECMP ECMP

AS65301

AS65302

AS65303

AS65401

AS65402

AS65403

AS65000

AS65200 AS65202AS65201 AS65203

cumulusnetworks.com

Automating the Clos Topology

Many switches to configure Automation is the same for 10 switches or 100 switches Same automation for switches and hosts

Want cookie-cutter configuration• As little node specific variation as possible

Cumulus Quagga BGP unnumbered configuration is very simple

cumulusnetworks.com

BGP Unnumbered with Cumulus Quagga

Cumulus uses RFC 5549 for BGP unnumbered

Cumulus greatly simplified neighbor statements

Makes automation super easy

router bgp 65200 bgp router-id 172.19.1.1 neighbor CUMULUS peer-group neighbor CUMULUS remote-as external neighbor CUMULUS timers 1 3 neighbor swp1 interface peer-group CUMULUS neighbor swp2 interface peer-group CUMULUS neighbor swp3 interface peer-group CUMULUS! address-family ipv4 unicast network 172.19.1.1/32!

leaf

spine

RFC 7404

cumulusnetworks.com

RFC 5549 in Action

leaf01# sh ip routeCodes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, P - PIM, T - Table, v - VNC, V - VPN, > - selected route, * - FIB route

K>* 0.0.0.0/0 via 192.168.0.254, eth0B>* 172.16.0.1/32 [20/0] via fe80::4638:39ff:fe00:5c, swp1, 00:08:03B>* 172.16.0.2/32 [20/0] via fe80::4638:39ff:fe00:2b, swp2, 00:08:03B>* 172.16.0.3/32 [20/0] via fe80::4638:39ff:fe00:3c, swp3, 00:08:03

C>* 172.16.1.1/32 is directly connected, loB>* 172.16.1.2/32 [20/0] via fe80::4638:39ff:fe00:5c, swp1, 00:08:03 * via fe80::4638:39ff:fe00:2b, swp2, 00:08:03 via fe80::4638:39ff:fe00:3c, swp3, 00:08:03

B>* 172.16.1.3/32 [20/0] via fe80::4638:39ff:fe00:5c, swp1, 00:08:03 * via fe80::4638:39ff:fe00:2b, swp2, 00:08:03 via fe80::4638:39ff:fe00:3c, swp3, 00:08:03

cumulusnetworks.com

BGP Unnumbered with Ansible- Example Template File

cumulusnetworks.com

Cumulus Network Command Line Utility

Configure directly from bashGuardrails includedEmbedded help/examples includedRollback supported

cumulus@leaf01:~$net add bgp autonomous system 65200cumulus@leaf01:~$net add bgp router-id 172.19.1.1cumulus@leaf01:~$net add bgp network 172.19.1.1/32cumulus@leaf01:~$net add bgp neighbor swp1-3 interfacecumulus@leaf01:~$net add bgp neighbor swp1-3 remote-as external

leaf

spine

https://cumulusnetworks.com/blog/cumulus-linux-network-command-line-utlility/

cumulusnetworks.com

Cumulus Quagga Logging

Logs: log file /var/log/quagga/quagga.log

sudo journalctl -f -u quaggaOct 28 21:31:44 leaf01 quagga[1076]: Starting Quagga monitor daemon: watchquagga.Oct 28 21:31:44 leaf01 quagga[1076]: Exiting from the scriptOct 28 21:31:44 leaf01 watchquagga[1130]: watchquagga 0.99.24+cl3eau5 watching [zebra bgpd ], mode [phased zebra restart]Oct 28 21:31:45 leaf01 watchquagga[1130]: bgpd state -> up : connect succeededOct 28 21:31:45 leaf01 watchquagga[1130]: zebra state -> up : connect succeeded

2016/11/03 16:49:26.613476 BGP: %ADJCHANGE: neighbor swp1 Up2016/11/03 16:49:26.613527 BGP: %ADJCHANGE: neighbor swp2 Up2016/11/03 16:49:26.613545 BGP: %ADJCHANGE: neighbor swp3 Up

cumulusnetworks.com

Troubleshooting BGP

Show ip bgp summaryleaf01# show ip bgp summaryBGP router identifier 1.1.1.1, local AS number 65001 vrf-id 0BGP table version 2RIB entries 5, using 640 bytes of memoryPeers 2, using 42 KiB of memory

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcdspine01(swp1) 4 65000 99 100 0 0 0 00:04:37 1spine02(swp2) 4 65000 46 48 0 0 0 00:02:02 1spine03(swp3) 4 65000 87 88 0 0 0 00:01:04 1

Total number of neighbors 3

leaf01# show ip bgp nei spine01BGP neighbor on swp1: fe80::4638:39ff:fe00:5c, remote AS 65000, local AS 65001, external linkHostname: spine01 BGP version 4, remote router ID 10.10.2.1[snip]

cumulusnetworks.com

Troubleshooting BGP

cumulus@leaf01:~$ sudo tail -f /var/log/quagga/quagga.log 2016/11/04 15:29:13.802083 BGP: %NOTIFICATION: received from neighbor swp2 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes fd e82016/11/04 15:29:13.802131 BGP: %NOTIFICATION: received from neighbor swp1 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes fd e82016/11/04 15:29:14.019099 BGP: %NOTIFICATION: sent to neighbor swp2 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes fd e82016/11/04 15:29:14.019254 BGP: %NOTIFICATION: sent to neighbor swp1 2/2 (OPEN Message Error/Bad Peer AS) 2 bytes fd e8

Debug bgp neighbor-events

cumulusnetworks.com

Key Takeaways for eBGP in a Data Center

eBGP works well as DC routing protocol

eBGP fits Clos topology well

eBGP unnumbered simplifies

cumulusnetworks.com

Network Virtualization Technologies, such as VXLAN can create Layer 2 Overlays over the Layer 3 fabric

Openstack requires VXLAN for server communications over Layer 3

Fitting Existing Applications in Layer 3 Fabric

leaf

spine

cumulusnetworks.com

Solution Deployment

cumulusnetworks.com

Dell EMC Open Networking

Optional 3rd party SDN/ NVO solutions

Standard orchestration & automation tools

Any networking OS

Open standard hardware

Merchant siliconON Switches

OrchestrationAutomation Monitoring

NetOpsDevOps

OS10

Software defined data center through open/disaggregated networking

cumulusnetworks.com

Project Inventory

Compute:–3 DELL EMC R220 Controller Nodes–300 Dell EMC R220 Compute Nodes–1 Dell EMC R630 as Director/Undercloud NodeNetwork:–6 Dell EMC S6010-ON switches for SPINE–18 Dell EMC S4048-ON switches for LEAF–Cumulus Linux –Cumulus Quagga Linux PackageOpenstack Distribution:–Red Hat Openstack Platform 7

cumulusnetworks.com

Deployment Topology

Layer 3 Networking throughout with Cumulus Linux Routing on the Host with Cumulus Quagga on all Compute Nodes Dell EMC Open Networking switches with ONIE

Configuration verified with Virtual prototype using Cumulus VX

Config Automation with Ansible

ECMP

ECMPECMPECMPECMP

Layer 3 Domain

cumulusnetworks.com

Deployment with ZTP, Ansible and Platform Director

Cumulus Linux deployment using Zero Touch Provisioning (ZTP) on all SPINE/Leaf Switches

Deploy Switch Configuration with Ansible Playbook on SPINE and LEAF switches

Deploy Cumulus Quagga with Ansible on all Compute Nodes and Controller Nodes and configure to join L3 fabric

Deploy OpenStack with Redhat Openstack Platform Director

ECMP

ECMPECMPECMPECMP

Layer 3 Domain

cumulusnetworks.com

Automation with Ansible

cumulusnetworks.com

Automation with Ansible Playbook

• Run geninv.sh to generate inventory and bootstrap host files • Run bootstrap.yml to create mgmt. network• Run site.yml to deploy playbook

./geninv.sh

Ansible-playbook –i bootstrap bootstrap.yml

Ansible-playbook –i inventory site.yml

Undercloud Server:Ansible Controller and OSP Director**

**OpenStack Platform Director facilitates planning, deployment and on-going operations of RHEL OpenStack Infrastructure

OOBM

cumulusnetworks.com

Automation with Ansible

Bgpd.conf.j2: Jinja2 templateQuagga configuration template

Site.yml: main Ansible playbookAutomates network deployment

Bootstrap.yml: Ansible playbookAutomates overcloud mgmt. network

cumulusnetworks.com

Deployment Results

• 100% Linux in the entire Rack• 15 minutes to deploy Switch

Configurations with Ansible playbook

• Less than 6 hours to build overcloud with Redhat Openstack Platform Director

• Stress test with Rally and analyze with Browbeat

ECMP

ECMPECMPECMPECMP

Layer 3 Domain

cumulusnetworks.com

OpenStack Rally

Rally Sample Report(s)

cumulusnetworks.com

Openstack Deployment Benefits with Routing on the Host

Keeps the Network Simple Only using Layer 3 Routing

Advertise loopbacks onlyNo ML2 Driver needed on the switches

VXLAN VTEP created Host to Host through Neutron

cumulusnetworks.com

Thank You!

© 2015 Cumulus Networks. Cumulus Networks, the Cumulus Networks Logo, and Cumulus Linux are trademarks or registered trademarks of Cumulus Networks, Inc. or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. The registered trademark Linux ® is used pursuant to a sublicense from LMI, the exclusive licensee of Linus Torvalds, owner of the mark on a world-wide basis.