View
9
Download
0
Category
Preview:
Citation preview
NANOG-77 [2019]
SONiC at Comcast Datacenters(Software for Open Networking in Cloud)
Think Outside of Cloud
- Alankar SharmaSr. Principal Architect, DC NetworkingComcast
- Vincent CelindroGlobals Architect - Networking
1
Agenda
Ø Comcast Datacenter Network OverviewØ Current Challenges & Motivation for New NOS: SONiCØ What is SONiC?
§ BYO Container Apps§ SONiC Subsystem§ BGP Update Processing
Ø SONiC MaturityØ SONiC BenefitsØ SONiC at Comcast
§ Current State of Deploymento Featureso Roadmap
Ø Comcast DC Quad DesignØ DC Core running SONiCØ Monitoring & Management
§ Telemetry & Data Sources§ Grey Failures: ML Based Anomaly Detection
Ø Advanced Features: Fast Reboot CLI Demo2
Comcast DC Network Overview
• 8 National DC
• 14 Regional DC
• 54 Local DC
• 8 Syndication DC
• 4 Video Distribution DC
• + POPs and Head Ends
• 9500+ Network device
• 300,000 server ports
• 600+ Changes/mo
• Multi Vendor: 8+ vendors for n/w infra
• 117 Hardware Models
• 184 versions of NOS Images
• MGMT Tools- (you name it!) -CVP, JunOS,
DCNM, HPNA, TAIL-f, Home Grown
Complex Eco-systemLarge Footprint
3
Comcast DC Network Overview
• Heterogeneous Infra types- Various Cloud Platforms (OpenStack, VMW, CaaS,Cloud Foundry), Storage Systems, Voice Systems, Baremetal Servers, VideoSystems, Appliances, Legacy Appliances
• Applications: Video, Multicast, DVR, Encoders, Big Data, Voice, HSD, CDN, X1
• Security (Permissions, Access Control, Asset protection, Copyrights), PCI/PIIZones- DMZ, Internal, PCI Sensitive, Office (IT)
• Unlike web-scaler cookie-cutter, but more customized, siloed deployments duevariety of applications and appliances
• Virtualized but considerable Baremetal servers
4
Why a new NOS at Comcast? Our MotivationWhat if the whole world spoke just English…
• Automation Challenges: Needed to build an abstraction layer that can consume common templatesand services. Attempts were made with Tail-f and OpenConfig
• Normalize the hardware variations. Empower to share same configuration, provisioning and mgmttools
• Minimize the software interop issues, even with same vendor
• Reduce cost. Pay only where you need to. Lean DC core and features pushed down to the edge.(Avoid any bandwidth-based NOS charges)
• Break-free: No more hostage to vendors. Feature Velocity and portability
• Multi-Vendor and Chip Diversity
• Decouple not only the HW/SW, also the SW/SW (containers)
• Commercially supported but easy to pull the plug, unlike proprietary NOS. And no perpetual licensebaggage.
5
What is SONiC?
Software for Open Networking in Cloud
• Linux software and integration.
Full interaction with native Linux shell
• Strong and growing support from
ODM and ASIC vendors
• SAI (Switch Abstraction Interface):
Hardware independent
• Containerized Network Apps
• Advanced management tools
Switch Abstraction Layer (SAI)
SWSS Utility
DB Platform LLDPTeamD
SNMP DHCP IPv6 BGP LAG Apps
Management Tools
Ansible K8S Jenkins CHEF PuppetDockerSwarm
SONiC
SyncDRedisDB
6
BYO Container Apps
• Components Isolation
• Select Building blocks
• Easy deployment
• Transactional
Benefits• Serviceability
• Extensibility
• Development agility
Database Platform
SWSS Utility
TeamD LLDP
RedisDB SyncD
SNMP DHCP BGP IPv6 LAG
Quagga FRR GOBGP Arista BGP
Juniper CRPD
SONiC Base
Network Apps
BYO
7
SONiC Subsystem
Fans Transceivers ASICLED Power Sensors
Platform Drivers Network Drivers ASIC Drivers
portsyncdintfsyncd
neighsyncd
orchagentintfmgrd
vlanmgrd
dhcprelay
Redis Server
SWSS container DB container
sensordfancontrol
smnpdsnmp agent
teamdteamsyncd
bgpdzebrafpmsyncd
lldpdlldpmgrdlldpsyncd
syncdsai apiasic sdk
CLIConfig-gen
Kernel Space
Hardware
User Space
Syncd container
DHCP PMON SNMP LLDPBGPTeamD
8 Ref Link: https://github.com/Azure/SONiC/wiki/Architecture
BGP Routing Innerworkings Example
Ref Link: http://azure.github.io/SONiC/
1. Neighbor sends new BGP prefix as an update,which is received at BGP's socket in kernel
2. BGP socket propagates the update to the bgpdprocess
3. BGPd interprets the update and notifies Zebraof the new route
4. Zebra validates the new route and delivers thenetlink-route message to fpmsyncd
5. fpmsyncd updates the APP DB6. Orch Agent, subscribes to APP DB, realizes the
state change7. Orch Agent, processes the updates of the App
DB and do corresponding changes in the SAIDB via SAI Redis.
8. Syncd receives the new state, generated by theorchagentd
9. Syncd uses SAI API to push the route into ASICvia the ASIC driver
1
2
3 4
56
78
9
9
SONiC Maturity
• Community Ecosystem• 3 releases per year• 120-250 commits/months• ~850 community members• ~200 active code contributors• 68+ supported platforms
• Monitoring Tools Available• Proven Management Tools
• Major Customers are running it in Prod• Contributing & Hardening• Ali Baba, Tencent, Linkedin, Dell,
Broadcom, Mellanox
• Commercial Support Available
BGP
ECMP
VLANTrunki
ng
LLDP
QoS
FlowContro
l
WRED
SNMP
Syslog
NTP
Mirroring
DHCP
VLAN
ACL
IPv6
TunnelDecap
COPP
BGP Gracef
ul
BGP MP
FasrReboo
t
TACACS
LAG
ConfigDB
MACAging
Critical Res.
Monitoring
FIBAccn
RDMA
gRPC
AsymmetricPFC
PFCWatermark
WarmReboo
t
VRFVxLA
N
PMON
FRR
BGPEVPN
Dtel
10
SONiC Advantages
• Designer NOS: Operators can select best building blocks. Containerization gives choiceof per component level. E.g. BGP- FRR, Quagga, GOBGP, Arista BGP, Juniper CRPD
• Not everyone has high negotiation power. SONiC is same cost for all, favoring small-midsize companies
• No carry forward baggage from monolithic codes
• Dedup: All NOS vendors are building same L2/L3 stack
• Zero downtime upgrade through docker swarm, patches in hours and new features rolledout quickly
• Hitless upgrades: Upgrade thousands of switches in a day• Fast Reboot, Warm Reboot
• CI/CD practices
11
SONiC Comcast Deployment
• Deployed in DC Lean Core (v4/v6 L3 underlay)• Team: Five engineers + Ops (24/7) as the team to manage/deploy • Plans to expand into Leaf Layer
• Deployed FeaturesBGP, ECMP, LAG, SYSLOG, LLDP, DHCP, TACACS, CoPP, IPv6, Fast Reboot, Everflow, Telemetry, Warm Reboot, ACL, NTP(Unlike other vendor NOS with monolithic codes)
• Roadmap (Testing)VLAN, VLAN Trunk, VxLAN, VRF, MLAG, sFLOW, BGP EVPN, INT
• DesiredBGP Unnumbered
12
Comcast Datacenter (HLD)DC Quad Design
DC Core
SAS01 SAS02
Nx100G Nx100G
AS01 AS02 AS03 AS04
S01 S02
sw001a
sw001b
sw001c
Server(n)
mgmt
Backbone
S(x) S(x+1)
Sw(n)csw002
asw002
b
Server01
sw002c
Server(n)
mgmt
sw002a
sw002b
sw002c
Server(n)
mgmt
sw(n)a sw(n)b
Server01
sw(n)c
Server(n)
mgmt
sw001a
sw001b
Server01
sw001c
Server(n)
mgmt
sw002a
sw002b
Server01
sw002c
Server(n)
mgmt
sw002a
sw002b
Server01
sw002c
Server(n)
mgmt
sw(n)a sw(n)b
Server01
sw00(n)c
Server(n)
mgmt
SPINE65003
LEAF65004
AGG SPINE65002
SUPER AGG SPINE65001
EBGP
EBGP
EBGP
EBGP
L3P2P
L3P2P
L3P2P
cs(x)a cs(x)b
sw(x)c
Server
mgmt
LBNAT
PolicySecurity
Default Route
Specific Routes
Specific Routes
Default Route
Default Route
Specific Routes
Default Route
Specific Routes
Server01 Server01
S(n) S(n+1)
13
SONiC in DC Core
• T2S (Tier 2 Spine) = TH3, 32x400G• S (Spine) = TH3, 32x400G• 204.8T Non-Blocking Fabric• 2048 x 100G ports for Leaf Switches• Lean Core, Underlay Only• 40% Cost Reduction• Standardize Optics (DR4, SM, 500m)• Low Latency
L3 P2PEBGP
400G400G
T2S01 T2S02 T2S08
S01 S02 S15 S16 S31 S32
T2S09 T2S16T2S15
1
2
3
4
S01
S16
T2S01 S17
S32T2S16
S01
S16
T2S01 S17
S32T2S16
S01
S16
T2S01 S17
S32T2S16
S01
S16
T2S01 S17
S32T2S16
14
SONiC Monitoring and Management
• Augtera Networks provides AI based network monitoring with SONiC integration
• Machine learning based real-time anomaly detection and auto-correlation along with notifications
• Ad-Hoc queries on real-time and historical data to troubleshoot incidents and analyze impact to customers / applications
• On demand or AI triggered flow collection and analytics leveraging Everflow*
Data collection from SONiC
Comcast Data Center
ML Anomalies & Incidents
Augtera AI Network Pulse
DC Ops & NOC Tools
SAS01 SAS02
Nx100G Nx100G
AS01 AS02 AS03 AS04
S01 S02
sw001a sw001b
Server01
sw001c
Server(n)
mgmt
S(n) S(n+1)
Backbone/ISP
S(x) S(x+1)
Sw(n)csw002a sw002b
Server01
sw002c
Server(n)
mgmt
sw002a sw002b
Server01
sw002c
Server(n)
mgmt
sw00(n)a
sw00(n)b
Server01
sw00(n)c
Server(n)
mgmt
sw001a sw001b
Server01
sw001c
Server(n)
mgmt
sw002a sw002b
Server01
sw002c
Server(n)
mgmt
sw002a sw002b
Server01
sw002c
Server(n)
mgmt
sw00(n)a
sw00(n)b
Server01
sw00(n)c
Server(n)
mgmt
SPINE65003
LEAF65004
AGG SPINE65002
SUPER AGG SPINE65001
EBGP
EBGP
EBGP
EBGP
Default Route
Default Route
Default Route
Default Route
L3P2P
L3P2P
L3P2P
Specific Routes
Specific Routes
Specific Routes
Specific Routes
cs00(x)a cs00(x)b
sw00(x)c
Server
mgmt
LB
NATPolicy
Security
AI & Operator Driven Config Changes
15
Telemetry
Data Analytics
SONiC Switch
SNMP CLI Syslog gRPC Mirror Everflow
Data Collection
Ops Monitoring Troubleshooting
16
SONiC Data Sources LeveragedData Type API Type NotesNetwork Topology SNMP L2 and L3
System Events Syslog
Control Plane State & Metrics
SNMP Comcast is using BGP
*Sensor data
(Roadmap)
Sonic GRPC based Telemetry streaming
CPU, temperature, fan speed etc.
Port counters SonicTelemetry streaming inPackets128To255Octets ,inOversizePacket, outOversizePackets, inBroadcastPackets, ifInDiscards, ifInErrors,
inMulticastPackets, ifHCInOctets, inUnicastPackets, inUnknownProtos, outBroadcastPackets, ifOutDiscards, ifOutErrors,outMulticastPackets, ifHCOutOctets, outUnicastPackets
Queue counters Sonic Telemetrystreaming queueStatPackets, queueStatBytes, queueStatDroppedPackets, queueStatDroppedBytes
*Buffer Statistics and Tracking (BST)
(Roadmap)
SonicTelemetry Streaming
Coming soon
Buffer count and queue entries for port-priority-group, service-pool, port-service-pool and queue categoriesTop-drops, top-port-queue-drops, port-drops, port-queue-drops
*Packet Telemetry
(Roadmap)
Sonic Everflow On demand and AI triggered
17
ML Based Anomaly Detection with SONiC
• Grey failures and traffic anomalies detection using Augtera AI Network Pulse
• No thresholds configured (e.g., BST thresholds)
-ML learns the patterns and automatically triggers anomalies on anomalous queue utilization or drops
• Congestion detection benefits from existing and additional counters in the roadmap
-Port counters: Finds anomalies on traffic, discards, oversize packets and 128To255Octet packets
-Queue counters and BST metrics are in progress
18
CLI/Demo Screenshots
• Fast Reboot – Minimal Disruption (30 Seconds)• Warm Reboot – Sub-second Disruption
19
DEMO Setup
sjc-z9100-01
server37
server39
sjc-z9100-02150.0.0.0/31
.0 .1
100.0.0.0/24 200.0.0.0/24
.1 .1
.2 .2
iperf
Fast-reboot DUT
20
Setup in Default State- iperf initiated from window 1
1
21
Fast-reboot in progress
Control plane goes for reboot with no disruption to data plane traffic (Window 1, 3)
1 2
3 4
22
Warm reboot in progressData plane rebooted in 15s after control plane is up
Only 15 second data plane time out
23
Links & References
• Command-Referencehttps://github.com/Azure/sonic-utilities/blob/master/doc/Command-Reference.md
• Virtual Sonic setuphttps://github.com/Azure/sonic-mgmt/blob/master/ansible/doc/README.testbed.VsSetup.md
• Mailing Listsonicproject@googlegroups.com
• Githubhttps://github.com/Azure/SONiC/
• Wikihttps://github.com/Azure/SONiC/wiki
• SAIhttps://github.com/opencomputeproject/SAI
24
SONiC at Comcast Datacenters
- Alankar Sharmaalankar_sharma@comcast.com
- Vincent CelindroNetwork /R/evolutionist
25
Thank You
Recommended