BRKARC-2003
Cisco ASR 9000 Architecture
Dennis Cai, ERBU Technical Marketing Architect
CCIE # 6621
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 2
Session Goal
To provide you with a thorough understanding of the Cisco ASR 9000 Router architecture, RSP, fabric, and line card design, packet flows, and key forwarding engine functions and features
This session will not examine baseline IOS-XR, for example, IOS-XR control plane and management plane protection, modular OS design, software package, SMU installation, IOS-XR routing configuration, etc. Please refer to TECARC-2001 (Cisco IOS-XR - Cisco Next Generation Operating System) for IOS-XR related technical information
2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 3
Building Carrier Ethernet Services Using Cisco Ethernet Virtual Circuit Framework - BRKSPG-2204 (EVC SW Infrastructure)
Redundancy Mechanisms for Carrier Ethernet and Layer 2 VPN Services -BRKSPG-2207 (L2VPN Resiliency)
Deploying Next Generation Carrier Ethernet: Services, Architectures and Operations - TECOPT-2100 (Carrier Ethernet design)
End-to-End Video Architecture and Design Part 1: Video Fundamentals, HeadendDesign, and Video Optimized Transport - TECSPV-1001 (Video solution)
End-to-End Video Architecture and Design: Part 2 - Access Networks and Video Service Delivery - TECSPV-1002 (Video solution)
Next Generation Assurance for IP Video Delivery Networks - BRKSPV-2111(video inline morning and management)
Deployment challenges with Interconnecting Data Centers - BRKDCT-3060 (Data center interconnect)
And more …
Related Sessions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 4
ASR 9000 Functional EvolutionNext-Generation SP Edge and Aggregation, DC Edge
2012
Mark
et F
un
cti
on
ali
ty
High Scale
Ethernet
Subscriber
Awareness
2011
Videoscape & IPv6
Advanced
Services
Available
Layer 2
Carrier
Ethernet
Rich L3 VPN Services
& Legacy Interfaces
IP RAN Backhaul
For Ethernet & TDM
Mobile
Backhaul on
ATM & CEoPS
Application
Service
Integration
Broadcast &
On-Demand
Video Delivery
Data Center
PE &
Interconnect
Network
Virtualization
(nV)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 5
ASR 9000 Portfolio Evolution NxDS0 – 120 Gbps LC
TODAY
# of Slots 10 slots 6 slots
Max. Linecards per
Chassis8 LC + 2 RSP 4 LC + 2 RSP
Max. Linecard Bandwidth 120 Gbps 120 Gbps
Max. Slot Bandwidth 180 Gbps 180 Gbps
Chassis Bandwidth 2.8 Tbs 1.4 Tbs
ASR 9010 ASR 9006
2H 2011
# of Slots 10 slots 6 slots
Max. Linecards per
Chassis8 LC + 2 RSP 4 LC + 2 RSP
Max. Linecard Bandwidth 240 Gbps 240 Gbps
Max. Slot Bandwidth 440 Gbps 440 Gbps
Chassis Bandwidth 6.4 Tbs 3.2 Tbs
1H 2012
# of Slots 22 Slot 10 Slot 6 slots Pizza Box“Iron Man”
9000v
Satellite
Max. Linecards per
Chassis20 LC 8 LC 4 LC 2 IO Slots Fixed Ports
Max. Linecard
Bandwidth360 Gbps 360 Gbps 360 Gbps
Modular +
4x10GE
44xGE +
4x10GE
Max. Slot Bandwidth 1.2 Tbps 440 Gbps 440 Gbps
Chassis Bandwidth 48 Tbps 6.4 Tbps 3.2 Tbps 120 Gbs 80 Gbs
NxDS0 – 240 Gbps LCNxDS0 – 360 Gbps LC
Iron ManASR 9922
ASR 9000v
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 6Foundation for Next-Generation Edge
Increased Service
VelocityQuickly deploy new
services
Multi-dimensional
ScaleSystem and services
scale
ASR 9000 “nV System”
ASR
9000v
ASR 9000
ASR
9000v
nV
Satellite
nV
clusterNetwork
Cloud
Client
Simplify OperationsReduce overall TCO
Integrated A to Z
Management
ASR 9000 Network Evolution – nV TechnologySuper, Simple
http://www.cisco.com/web/solutions/sp/asr9000.html
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 7
Agenda
Hardware Architecture
‒ Chassis, RSP, Line Cards and Switch Fabric
Software Architecture
‒ Packet Forwarding Overview
‒ L3 Forwarding
‒ L2 Forwarding
‒ Load Balancing (BGP Multi-path, ECMP, Link Bundle)
‒ HA and Fast Convergence
‒ QoS Architecture
Data Plane Troubleshooting
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 8
ASR 9010 and ASR 9006 ChassisIdentical HW components across two chassis*
Integrated cable
management
with cover
RSP (0-1)
Line Card
(0-3, 4-7)
System fan trays
Six Modular
Power Supplies
Front-to-
back airflow
System fan trays
Three Modular
Power Supplies
RSP (0-1)
Line Card
(0-3)
Air draw
Side-to-back
airflow
cable
management
* Fan tray is different
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 9
Power and CoolingExisting Power Supply and Fan are ready for 400G/slot
2.1/1.5 kW
2.1 kW
3 kW
DC Supplies
AC Supplies
AB
AB
3 kW
A
B
Fans unique to chassis
Variable speed for
ambient temperature variation
Redundant fan-tray
Low noise, NEBS and OSHA compliant
6 & 10 slot use same power supplies
Single power zone
All power supplies run in active mode
Power draw shared evenly
50 Amp DC Input or 16 Amp AC
for Easy CO Install
ASR 9010 Fan Tray
Power Supply
ASR 9006 Fan Tray
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 10
Useful CLIs
RP/0/RSP0/CPU0:asr1(admin-config)#power budget reservation standby-rsp disable to
disable power reservation for secondary RSP if only single RSP in the system
RP/0/RSP0/CPU0:asr1(admin-config)#power budget enforcement disable to disable the
power budget enforcement. Be very cautious, if system run out of power under certain condition,
it may cause undeterminstic state
RP/0/RSP0/CPU0:asr1(admin)#show environment power-supply
<snip>
Power Shelves Type: AC
Total Power Capacity: 3000W
Usable Power Capacity: 3000W
Supply Failure Protected Capacity: 0W
Worst Case Power Used: 2200W
Slot Max Watts
---- ---------
0/RSP0/CPU0 235
0/RSP1/CPU0 235
0/0/CPU0 630
0/1/CPU0 350
0/FT0/SP 375
0/FT1/SP 375
Worst Case Power Available: 800W
Supply Protected Capacity Available: Not Protected
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 11
Available power is checked when:
–An LC card is inserted
–An LC card is powered up via the CLI
–An LC card is reset via “hw-mod reload”
If the system does not have enough available power to accommodate the LC, then the LC becomes “UNPOWERED”
Installing new power supplies will not automatically power up any UNPOWERED line cards. The user can force a recheck using:
“hw-mod reload loc <>”
RSP and Fan Tray cards are given priority allocation of power budget
LC power budget is checked in numeric order until it is exhausted. The actual power up of the LC's is in parallel, and asynchronous
Power Check and Rules
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 12
RSP Engine
Performs control plane and management functions
Dual Core CPU processor with 4GB or 8GB DRAM (4G8G is not field upgradable)
2MB NVRAM, 4GB internal bootdisk, 2 external compact flash slots
Dual Out-of-band 10/100/1000 management interface
Console & auxiliary serial ports
Hard Drive: 70G HDD
Console Port
AUX PortManagement
Ethernet
ALARMBITS
Compact Flash
Slots
Status LED
Status light
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 13
Front Panel CPU Complex
RSP Engine Architecture
4/8GB MEM
I/O FPGA
HDD
Mgt Eth
CF card
Console
NVRAM Boot Flash
EOBC/Internal GE switch
EtherSwitch
Timing Domain
ClockTimeFPGA
BITS
4G disk
Mgt Eth
Aux
Alarm
Switch fabric
PuntFPGA Arbitration
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
FIA
CPU
Arbiter
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 14
RSP Engine ArchitectureFile system
RP/0/RSP0/CPU0:asr9k#dir ?
/all List all files
/ena Recognize sub directories
/recurse Recursively list subdirectories encountered
WORD file name
bootflash: bootflash: file system boot image
compactflash: compactflash: file system
compactflasha: compactflasha: file system
disk0: disk0: file system XR image, configurations
disk0a: disk0a: file system
disk1: disk1: file system
disk1a: disk1a: file system
harddisk: harddisk: file system SYSLOG, core dump
harddiska: harddiska: file system
harddiskb: harddiskb: file system
location Fully qualified location specification
nvram: nvram: file system ROMMON variables
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 15
RSP Operations Impact Fabric ?Guarantee “0” packet loss for RSP failover or OIR
Switch fabric ASIC reside on the RSP blade physically
Switch fabric ASIC is controlled by low level hardware, it operates separately from RSP function
All fabric ASIC run in active mode regardless of the RSP status
RSP SW switch over, reload, crash including kernel crash have NO impact on fabric operation
RSP OIR has no traffic impact due to long/short pin backplane design and instant fabric switch over
–Short pin trig the control signaling for fabric switchover in hardware
–Long pin is used for data packet. It can continue draining the in-flight packets from the fabric during the extended short period of time
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 16
Ethernet Line Card FamilyShipping
A9K-40G A9K-4T A9K-8T/4 A9K-2T20G A9K-8T A9K-16T/8
Common HW ASIC and HW architecture
Identical SW features
Line card Memory options for QoS scale
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 17
NP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA0
CPU
NP4PHY
NP6PHY
NP7PHY
NP5PHY
FIA1
Example: A9K-8T
B0
NP: Network ProcessorMain forwarding ASIC
L2 & L3 forwarding, features (QoS, ACL, etc),
control plane policing, mcast replication, etc
10Gbps bi-directional with features applied
CPU (same as RSP)Program HW forwarding tables
Distributed Control planesSW switched packets
Inline Netflow
FIA: Fabric Interface ASICProvide non-blocking data connection to switch fabric
Internal system queues/VoQIntelligent mcast replication
B: Bridge FPGAProvide non-blocking data
connection between NP and FIAInternal System queues
Intelligent mcast replication
10G PHY for one or two 10G port, or
10x1G port
Line Card Architecture – Hardware Components
Note, Bridge FPGA provide non-blocking
connection between NP and the FIA. Functionally it
does the HW conversion due to different interface
format on NP and FIA. It‟s part of the switch fabric
connection. To make it logically simple, it will be
removed from the remaining slides.
B1
B0
B1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 18
RP/0/RSP1/CPU0:SJC#show controllers fabric ?
Arbiter Arbitration ASIC show screens.
Crossbar XBAR ASIC show screens.
fia Show command for fabric interface asic
RP/0/RSP1/CPU0:SJC#show controllers fabric fia bridge stats location 0/0/cpu0
RP/0/RSP1/CPU0:SJC#show controllers fabric fia stats location 0/0/cpu0
RP/0/RSP1/CPU0:SJC#show controllers np ?
counters Display contents of global stats counters
crashinfo Display NP Crash info
drvlog Display Driver Logging
fabric-counters XAUI counters dump
interrupts Show NP interrupt data
memory NP Raw Memory Dump
portMap Show port mapping on NP
ports Shows physical ports associated with each np
<snip>
Line Card HW Components – Counters
FIA
NP
Crossbar
Fabric
ASIC
Arbiter
FIA B0
B0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 19
4xNPs Line Card Family
NP0PHY 3
NP2PHY 1
NP3PHY 0
NP1PHY 2
NP0PHY 3
NP2
NP3
NP1
PHY 7
PHY 2
PHY 6
PHY 1
PHY 5
PHY 0
PHY 4
NP0
NP2
NP3
NP1
NP0PHY
NP2
NP3
NP1PHY
A9K-4T-E/B/L
A9K-8T/4-E/B/L
A9K-2T20G-E/B/L
A9K-40G-E/B/L
Oversubscribed line card Up to
60Gbps (~57Gbps) bandwidth
FIA
FIA
FIA
FIA
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 20
8xNPs Line Card Family
PHY
PHY
PHY
PHY
PHY
PHY
PHY
NP2
NP1
NP0
NP6
NP5
NP4
NP7
NP3PHY
NP2
NP1
NP0
NP6
NP5
NP4
NP7
NP3
PHYPHY
PHYPHY
PHYPHY
PHYPHY
PHYPHY
PHYPHY
PHYPHY
PHYPHY
A9K-8T-E/B/L A9K-16T/8-B
Oversubscribed line card
Up to 120Gbps (~117Gbps)
bandwidth
FIA0
FIA1
FIA0
FIA1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 21
Line Card Memory OptionsDifferent Queue Scale for Different Deployments
3 memory options for each line card: Extended (or high queue),
Base (medium queue), Low (low queue)*
Different memory option has different QoS queue/policer scale, L2
sub-interface scale and ACL scale
All other system wide scale is the same across different type of
the line cards, including FIB, MAC address, Bridge-domain, L3
sub-interface, VRF, etc
All line cards have the same HW Identical features
Mixed different type of line cards are supported on the same
chassis with same system wide scale and identical features
Low
Medium
High
* A9K-16T/8 only have “B” option
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 22
L/B/E Line Cards – What’s the Difference?
-
Forwarding ASIC
STATS MEMORY
FRAME MEMORYLOOKUP
MEMORY
TCAM
FIB MAC
Each NPU has Four Main Associated memories TCAM , Search/Lookup memory , Frame/buffer memory and statistics memory
–TCAM is used for VLAN tag, QoS and ACL classification
–Lookup Memory is used for storing FIB tables, Mac address table and Adjacencies
–Stats memory is used for all interface statistics, forwarding statistics etc
–Frame memory is buffer memory for Queues
E/B/L line card have different TCAM , Stats and Frame Memory size, which give different scale number of the QoS queues and L2 sub-interfaces per line card
Lookup Memory is the same across line card s why?
–To support mix of the line cards without impacting the system wide scale including routing, multicast, MAC address, L3 interface, MPLS label space scale
NP complex
Same lookup memory size across different type of line
cards mix-and-match LCs, same system level scale
Different memory size for different QoS
queuing/policing, L2 EFP and ACL scale
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 23
Optics
TenGE – XFP OpticsLR, ZR, & ER
GE - SFP OpticsT, S, L, & Z
Standard, CWDM, & DWDM XFPs/SFPs/SFP+ available
IPoDWDM G.709 FEC/EFEC
Optics official support matrix:
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 24
A9K-SIP-700
Powerful and Flexible QFP Processor
• Flexible ucode Architecture for Feature Richness
• L2 + L3 Services: FR, PPP, HDLC, MLPPP, LFI, L3VPN, MPLS, Netflow, 6PE/VPE
Strong QoS
• 128+K Queues,
• 128+K Policers
• H-QoS
• Color Policing
Scalability
• Distributed control and data plane; 20G; 4 Bays
• L3 i/f, route, session, protocol – scaled up for MSE needs
High Availability
• IC-Stateful Switch Over capability,
• MR-APS
• IOS-XR base for high scale and reliability
SIP/SPA – Non-Ethernet Support
SPA support list:
http://www.cisco.com/en/US/partner/prod/collateral/routers/ps9853/data_sheet_c78-573452.html
SPA Support: ‒ChOC-12: T1 / T3, MLPPP, LFI, IC-SSO, MR-APS‒ChOC-3/STM1 + ChOC-12: DS0 / STM1 + ChOC-48: DS3 / E3 / STM4‒POS: OC3/STM1, OC-12/STM4, OC-48/STM16, OC-192/STM64
SPA Roadmap (2011):Ch T1/E1, Ch T3/E3, CEoPs, ATM
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 25
Dual-FIA 8xNPs
Linecard
FIA0
FIA1
RSP1
Single-FIA4xNPs
Linecard
Switch Fabric Overview
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
RSP0
4x23Gbps =92Gbps with dual RSP
2x23Gbps=46Gbps with single RSP
Active-active load balancing: Unicast: per-packet load balancing, Multicast: per (S,G) load balancing
Arbiter for fabric access control. Arbiter is in active/standby mode, which is controlled by low level hardware signalling
Frame format over fabric: super-frame, it can aggregate multiple small packet into a big sup-frame to improve the fabric throughput
23Gbps per fabric channel
8x23Gbps =184Gbps with dual RSP
4x23Gbps=92Gbps with single RSP
FIA
Arbiter
Arbiter
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 26
Dual-FIA 8xNPs
Linecard
FIA0
FIA1
RSP1
Single-FIA4xNPs
Linecard
Fabric RedundancyGuarantee “0” packet loss during RSP failover and OIR
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
RSP0
Separate data and arbitration(control) paths
All fabric data channels run in active mode for extra fabric bandwidth and instant fabric switch over
Both Arbiters work in parallel – both answer all requests, FIAs follow active Arbiter, FIAs switch to backup if needed instant control path switchover
Arbiter switchover is controlled by low level hardware signalling
FIA
Arbiter
Arbiter
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 27
NP0PHY
NP2
NP3
NP1
FIA0
CPU
NP4
NP6
NP7
NP5
FIA1
B0
System Internal BandwidthNO bottleneck inside the system
B1
B0
B1
15Gbps bi-
directional30Gbps and 25M pps
(combined ingress and egress)
PHY
PHY
PHY
PHY
PHY
PHY
PHY
RSP1
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Arbiter
RSP0
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Arbiter
Each FIA has one fabric channel
which is 23 Gbps bi-directional, to
each of the switch fabric ASIC
A9K-16T/8-B
~120Gbps raw bandwidth
A9K-8T/4
~60Gbps raw bandwidth
30Gbps bi-
directional
60Gbps bi-
directional
30Gbps bi-
directional
Use A9K-8T line card as example
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 28
Agenda
Hardware Architecture
‒ Chassis, RSP, Line Cards and Switch Fabric
Software Architecture
‒ Packet Forwarding Overview
‒ L3 Forwarding
‒ L2 Forwarding
‒ Load Balancing (BGP Multi-path, ECMP, Link Bundle)
‒ HA and Fast Convergence
‒ QoS Architecture
Data Plane Troubleshooting
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 29
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPUNP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA
CPU
Egress LCIngress LC
Switch Fabric
1
Ingress NP look up Get egress NP information (added into the NP/fabric
header), apply ingress features
Egress NP look up Get egress logical port, VLAN, MAC, ADJ information, etc for packet rewrite, apply egress
features
IOS-XR Two-Stage Packet ForwardingFully Distributed Forwarding on Line Cards
2
Packet is forwarded to the egress NP based on the information in the
NP/fabric header
Each line card has independent AIB only for local interfaces
Each line card has independent Interface DB for local interfaces
Both Ingress and Egress FIB – allows forwarding features to be independently applied on LCs
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 30
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPU
NP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA
CPU
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPU
LC2
LC3
LC1
IGMP joins
Multicast Source
IGMP joins
Switch Fabric
1
FPOE
2
3
MGID
Fabric Replicationreplicate single copy to
each LC which has IGMP join, based on FGID table
in switch fabric
FIA Replication replicate single copy to each NP which
has IGMP join, based on MGID table in FIA
NP Replication replicate single copy per each
receiver based on multicast FIB table
1 2 3
MFIB
Multicast Packet ForwardingIntelligent Multicast Replication
FGID – Fabric Group ID
MGID – Multicast Group ID
MFIB – Multicast Forwarding Information Base
Efficiency
Line rate
Scalable
Simple
MFIB
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 31
IGMP joins
NP0
NP2
NP3
NP1
FIAB0
B1
CPUPHY
PHY
PHY
PHY
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHY
FIA
CPU
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHY
CPU
LC2
LC3
IGMP joins
Multicast Source
Switch Fabric
1
FGID
Fabric Replicationreplicate single copy to
LCs which receive IGMP join, based on FGID table
in switch fabric
FIA Replication replicate single copy to NPs which
receive IGMP join, based on MGID table in FIA
NP Replication replicate copy per receiver based on
multicast FIB table
1
2
4
Intelligent Multicast ReplicationSwitch Fabric and Egress LC Replication
FGID – Fabric Group ID
MGID – Multicast Group ID
MFIB – Multicast Forwarding Information Base
B0
B1
FIAB0
B1
IGMP joins
Bridge Replication similar as FIA replication, single copy
to NP
3
2
2
3
3
4
MGIDMGID
MFIB1 0000000000
2 0000000001
3 0000000011
…
N 1111001111
hit
FGID=3
FPOE Table
1Kx10 bitmap representing
all combination of the
egress linecards
1 00
2 01
3 10
…
N 10
hitMGIDReplicate to the first
Bridge only
MGID Table in FIA
Each entry represent one
mroute, 2 bits indicating the
down stream ASICs
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 32
Initial lookup
Internal
Replication
Engine
Second lookup
From Fabric
To egress
port
NP
Initial Lookup
– Original packet is lookup in the MFIB/L2FIB returning the number of copies to be replicated on the given NPU, each copy per output logical interface
– Packet is sent to a special internal replication engine. HW chips replicates the packets and send the copy to the processing engine
Second Lookup
–Each replicated copy is processed. Egress feature like QoS, ACL is applied at this pass
Intelligent Mulitcast Packet ReplicationInside NP
14Gbps
replication
capacity per NP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 33
ASR 9000 Flexible Ethernet SW Infrastructure(“EVC” SW Infrastructure, BRKSPG-2204 )
VLAN tag local significant
Flexible VLAN
tag classification
Flexible VLAN
tag rewrite
Flexible
Ethertype (.1Q,
QinQ, .1ad)
Flexible service mapping and multiplexing. Support all standard based
services concurrently on the same port:Regular L3, L2 interface/sub-interfaceIntegrated L2 and L3 – IRB/BVIMixed L2 and L3 sub-interfaces on the same port
1
2
(H-)VPLS
EoMPLS PW
EoMPLS PW
EoMPLS PW
L3 SubI/F
X
Bridging
Bridging
Routing
IRB
Routing and Bridging
IRB
X
L2 or L3 sub-interfaces(802.1a/qinq/.1ad)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 34
L2VPN P2P
L2VPN P2P service configuration example
l2vpn
xconnect group cisco
p2p service1 local connect
interface gig 0/0/0/1.101
interface gig 0/0/0/2.101
p2p service2 VPWS
interface gig 0/0/0/3.101
neighbor 1.1.1.1 pw-id 22
p2p service3 PW stitching
neighbor 2.2.2.2 pw-id 100
neighbor 3.3.3.3 pw-id 101
EFP configuration example
Interface gig 0/0/0/1.101 l2transport
encapsulation dot1q 101 second 10
rewrite ingress pop 2 Symmetric
Interface gig 0/0/0/2.101 l2transport
encapsulation dot1q 101
rewrite ingress pop 1 Symmetric
Interface gig 0/0/0/3.101 l2transport
encapsulation dot1q 102-105
rewrite ingress push dot1q 100 Symmetric
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 35
CLI Comparison – L2VPN P2P
ASR 9000 7600
Local Connect interface GigabitEthernet4/1/0service instance 101 ethernet
encapsulation dot1q 101 second 10rewrite ingress tag pop 2 Symmetric
interface GigabitEthernet4/1/1service instance 100 ethernet
encapsulation dot1q 100rewrite ingress tag pop 1 Symmetric
connect eline-101 GigabitEthernet4/1/0 101
GigabitEthernet4/1/1 100
EoMPLS interface GigabitEthernet4/1/1
service instance 11 ethernet
encapsulation dot1q 101 second-dot1q 60-70
xconnect 10.0.0.3 101 encapsulation mpls
PW stitching l2 vfi tac-training point-to-point
neighbor 10.0.2.3 3001 encapsulation mpls
neighbor 10.0.2.2 3000 encapsulation mpls
[note] require BGP configuration if it‟s for inter-
AS
EFP configuration under interface
Including VLAN tag encapsulation, tag rewrite, Qo/ACL features, etc
Interface gig 0/0/0/1.101 l2transportencapsulation dot1q 101 second 10rewrite ingress tag pop 2 Symmetric
Interface gig 0/0/0/2.101 l2transportencapsulation dot1q 101rewrite ingress tag pop 1 Symmetric
Service configuration under “l2vpn”
l2vpnxconnect group cisco
p2p service1 local connectinterface gig 0/0/0/1.101interface gig 0/0/0/2.101
p2p service2 EoMPLSinterface gig 0/0/0/3.101neighbor 1.1.1.1 pw-id 22
p2p service3 PW stitchingneighbor 2.2.2.2 pw-id 100neighbor 3.3.3.3 pw-id 101
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 36
L2VPN Multi-Point
L2VPN MP service configuration example
l2vpn
bridge group cisco
bridge-domain domain1 local bridging
Interface gig 0/0/0/1.101
split-horizon group no bridging among same SHG
Interface gig 0/0/0/2.101
split-horizon group
bridge-domain domain2 vpls
Interface gig 0/0/0/1.101
Interface gig 0/0/0/2.101
vfi cisco
neighbor 192.0.0.1 pw-id 100
neighbor 192.0.0.2 pw-id 100
bridge-domain domain3 h-vpls
Interface gig 0/0/0/1.101
neighbor 192.0.0.3 pw-id 100 spoke PW
vfi cisco for core PWs
neighbor 192.0.0.1 pw-id 100 core PW
neighbor 192.0.0.2 pw-id 100
EFP configuration example
Interface gig 0/0/0/1.101 l2transport
encapsulation dot1q 101
rewrite ingress pop 1 Symmetric
Interface gig 0/0/0/2.101 l2transport
encapsulation dot1q 101
rewrite ingress pop 1 Symmetric
Interface gig 0/0/0/3.101 l2transport
encapsulation dot1q 102
rewrite ingress push dot1q 100 Symmetric
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 37
CLI Comparison – L2VPN MP Bridging
ASR 9000 7600
Local Bridging interface GigabitEthernet4/1/0service instance 101 ethernet
encapsulation dot1q 101-1000
bridge-domain 100 split-horizon
interface GigabitEthernet4/1/1
service instance 101 ethernetencapsulation dot1q 101-1000
bridge-domain 100 split-horizon
(H-)VPLS &
SVI based
EoMPLS
interface GigabitEthernet4/1/0service instance 2 ethernet
encapsulation dot1q 20
bridge-domain 20 split-horizon
interface GigabitEthernet4/1/1
service instance 2 ethernetencapsulation dot1q 20
bridge-domain 20 split-horizon
l2 vfi vpls-20 manual VPLS
vpn id 120
neighbor 10.0.0.1 encapsulation mpls core PWneighbor 10.0.0.2 encapsulation mpls no-split-
horizon spoke PW
neighbor 10.0.0.4 encapsulation mpls
interface Vlan20 (H-)VPLS
xconnect vfi vpls-20
interface Vlan20 SVI EoMPLS
xconnect 10.0.0.1 101 encap mpls
EFP configuration under interface
Including VLAN tag encapsulation, tag rewrite, Qo/ACL features, etc
Same as L2VPN P2P services
Service configuration under “l2vpn”
l2vpn
bridge group cisco
bridge-domain domain1 local bridging
Interface gig 0/0/0/1.101
split-horizon group
Interface gig 0/0/0/2.101
split-horizon group
bridge-domain domain2 SVI EoMPLS
Interface gig 0/0/0/1.101
Interface gig 0/0/0/2.101
neighbor 192.0.0.1 pw-id 100
bridge-domain domain3 vpls & h-vpls
Interface gig 0/0/0/1.101
neighbor 192.0.0.3 pw-id 100 spoke PW
vfi cisco core PWs under VFI config
neighbor 192.0.0.1 pw-id 100
neighbor 192.0.0.2 pw-id 100
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 38
L2VPN E-Tree Using Split-horizon
L2VPN E-Tree service configuration example
l2vpn
bridge group cisco
bridge-domain domain1
Interface gig 0/0/0/1.101
split-horizon group leaf port
Interface gig 0/0/0/2
split-horizon group leaf port
Interface gig 0/0/0/3.101 root port
Interface gig 0/0/0/4 root port
Leaf
RootLeaf
Root
MPLS
E-Tree Forwarding Rules:
• Root can talk with root
• Root can talk with leaf or leaf
can talk with root
• but leaf can‟t talk with leaf
BD
VFI
MPLS
Root bridge port
Leaf bridge port
neighbor 192.0.0.3 pw-id 100
split-horizon group leaf access PW
neighbor 192.0.0.4 pw-id 100 root access PW
vfi cisco vfi PW is root PW ONLY
neighbor 192.0.0.1 pw-id 100
neighbor 192.0.0.2 pw-id 100
Root
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 39
Integrated L2 and L3 Support – BVI vs. SVI
7600 SVI exampleASR 9000 IRB/BVI example
Interface gig 0/0/0/1.50 l2transportencapsulation dot1q 50rewrite ingress tag pop 1 Symmetric
Interface gig 0/0/0/2.50 l2transportencapsulation dot1q 40rewrite ingress tag pop 1 Symmetric
Interface bvi 20
ipv4 address 1.1.1.1 255.255.255.0
l2vpn
bridge group cisco
bridge-domain domain50
Interface gig 0/0/0/1.50
Interface gig 0/0/0/2.50
routed interface bvi 20
interface gig 1/2
switchport
switchport mode trunk
switchport trunk allow vlan 50-1000
interface GigabitEthernet4/1/0
service instance 2 ethernet
encapsulation dot1q 50
rewrite ingress tap pop 1 sym
bridge-domain 50
Interface vlan 50
ip address 1.1.1.1 255.255.255.0
vfi cisco
neighbor 192.0.0.1 pw-id 100
neighbor 192.0.0.2 pw-id 100
neighbor 192.0.0.3 pw-id 100
xconnect 192.0.0.3 100 en mpls
interface vlan 50
ip address 1.1.1.1 255.255.255.0
xconnect vfi cisco
l2 vfi cisco manual
vpn id 50
neighbor 192.0.0.1 encapsulation mpls
neighbor 192.0.0.2 encapsulation mpls
neighbor 192.0.0.3 encapsulation mpls no-split-horizon
VLAN is global
VLAN is local significant, no
relationship with BVI number
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 40
Mixed L2 and L3 on the Same Port
L2 trunk port, use SVI for L3
interface gig 1/2
switchport
switchport mode trunk
switchport trunk allow vlan 50-1000
Interface vlan 50
ip address 1.1.1.1 255.255.255.0
Native L2 and L3 sub-interface on the
same port
interface gig 0/0/0/1.50
encapsulation dot1q 50
ip address 1.1.1.1 255.255.255.0
Interface gig 0/0/0/0/1.51 l2transport
encapsulation dot1q 51rewrite ingress tag pop 1 Symmetric
l2vpn define L2 servicebridge group test
<snip>
7600/6500
ASR 9000
SVI SVI
Native L2 and L3 sub-interface support on the
same physical port
No such limitations of the L3 features on the SVI
L2 trunk with
SVI for L3
Native L2 + L3 port
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 41
CLI Comparison – EVC vs. L2 switchport “trunk”
Interface gig 0/0/0/1.50 l2transportencapsulation dot1q 50rewrite ingress tag pop 1 Symmetric
Interface gig 0/0/0/1.51 l2transportencapsulation dot1q 51rewrite ingress tag pop 1 Symmetric
…
Interface gig 0/0/0/1.1000 l2transportencapsulation dot1q 1000rewrite ingress tag pop 1 Symmetric
l2vpn
bridge group cisco
bridge-domain domain50
Interface gig 0/0/0/1.50
Interface gig 0/0/0/2.50
<snip>
bridge-domain domain1000
Interface gig 0/0/0/1.1000
Interface gig 0/0/0/2.1000
interface gig 1/1
switchport
switchport mode trunk
switchport trunk allow vlan 50-1000
interface gig 1/2
switchport
switchport mode trunk
switchport trunk allow vlan 50-1000
ASR 9000 (EVC)
configuration example Switchport trunk
configuration
Switchport trunk model is easy for the L2
trunk configuration, especially with hundreds or
thousands of VLANs
EVC per sub-interface model has advantage
of providing per VLAN features, local VLAN
significant, and no global 4K VLAN limit, etc
EVC model is complex for provisioning if per
VLAN feature is not required. CLI enhancement
is under investigation
VLAN is global
VLAN is local port
significant
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 42
Flexible VLAN Tag Classification
RP/0/RSP0/CPU0:PE2-asr(config)#int gig 0/0/0/4.100 l2transport
RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation ?
default Packets unmatched by other service instances
dot1ad IEEE 802.1ad VLAN-tagged packets
dot1q IEEE 802.1Q VLAN-tagged packets
untagged Packets with no explicit VLAN tag
RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation dot1q 100-200,205 ?
comma comma
exact Do not allow further inner tags
RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation dot1q 100 second-dot1q 10-20,35 ?
comma comma
exact Do not allow further inner tags
RP/0/RSP0/CPU0:PE2-asr(config-subif)#encapsulation dot1ad 20 dot1q 10-20 ?
comma comma
exact Do not allow further inner tags
Double tag
Single tag
Multiple tag
Range of tag
.1q and/or .1ad
Loose or exact match
Untagged
Default tag
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 43
Flexible VLAN Tag Rewrite
RP/0/RSP0/CPU0:PE2-asr(config)#int gig 0/0/0/4.100 l2transport
RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag ?
pop Remove one or more tags
push Push one or more tags
translate Replace tags with other tags
RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag pop ?
1 Remove outer tag only
2 Remove two outermost tags
RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag push ?
dot1ad Push a Dot1ad tag
dot1q Push a Dot1Q tag
RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag push dot1q 100 ?
second-dot1q Push another Dot1Q tag
symmetric All rewrites must be symmetric
RP/0/RSP0/CPU0:PE2-asr(config-subif)#rewrite ingress tag translate ?
1-to-1 Replace the outermost tag with another tag
1-to-2 Replace the outermost tag with two tags
2-to-1 Replace the outermost two tags with one tag
2-to-2 Replace the outermost two tags with two other tags
Pop tag 1 or 2
Push tag 1 or 2
Tag translation
1-1
1-2
2-1
2-2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 44
Flexible Service and VLAN Tag ClassificationDeployment Example – Consolidated DC Edge
VFI
MC-LAG
interface bundle-ethernet1.10 L3 sub for internet access
encapsulation dot1q 10
ipv4 address 1.1.1.1 255.255.255.0
interface bundle-ethernet1.20 L3 sub for L3VPN
encapsulation dot1q 20
vrf customer1
ipv4 address 20.1.1.1 255.255.255.0
interface bundle-ethernet1.100 l2transport VLAN range for DCI1
encapsulation dot1q 100-1000 selective QinQ
interface bundle-ethernet1.200 l2transport VLAN range for DCI2
encapsulation dot1q 1001-2000 selective QinQ
L2vpn
bridge group DCI
bridge-domain DCI1
interface bundle-ethernet1.100
vfi DCI1
neighbor …
bridge-domain DCI2
interface bundle-ethernet1.200
vfi DCI2
neighbor …
VFI VFI VFI
vPC
Data Center 1
DC2DC3
L3VPN
Internet
DCIt
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 45
ASR 9000 L3 “At-a-Glance”Extend IOS-XR L3/MPLS to the ASR9000
IOS-XR routing and MPLS Widely deployed, proven
stability and feature rich
Built-in HA NSF/GR & NSR
& SSO
Distributed control plan High Multi-dimensional control plane scale, for example 5000
BGPs + 10,000 BFDs + etcHierarchical FIB
structure and Fast FIB programming
Prefix independent fast network convergence
IPv6 ready Dual stack, 6PE/6VPE,
high scale, line-rate IPv6 forwarding performance
Distributed inline Netflow1M entries/LC , 100Kpps /LC, flexible sampling and IPv4/IPv6/MPLS feature
support Mature IOS-XR
L3 Routing & MPLS
SW Infrastructure
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 46
L3 Control Plane Overview
LDP RSVP-TEBGP
ISIS
OSPF
EIGRP
Static
FIB Adjacency
LC NPU
ARP
LSD RIB
AIB
SW FIB
LC CPU
RP
AIB: Adjacency Information Base
RIB: Routing Information Base
FIB: Forwarding Information Base
LSD: Label Switch Database
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 47
L3 Forwarding – Hierarchical Data Structure FIB and ADJ Tables Overview
RecursivePrefix Leaf
Table
NR PrefixLeaf Table
RecursiveLDI
NR LDI 0
NR LDI 1
…
NR LDI N
0 1 2 3 . . . … nVRF table
Search Tree
Leaf Table
(FIB table)
LDI/ADJ table
BGP PIC
TE-FRR PIC
Link bundle PIC
PIC: Prefix independent convergence
LAG: Link aggregation group
LDI: Load information
OutI/FProtected
TE adj
OutI/FBackupTE adj
OutI/F
OutI/F
LAG
LAG
adj
adj
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 48
Show CLI Examples –Validate the L3 Forwarding Path
show ip route 10.0.0.4
show cef 10.0.0.4/32 RSP‟s view
Show cef 10.0.0.4/32 location 0/2/cpu0 Local Line card‟s view
Show cef 10.0.0.4/32 detail location 0/2/cpu0detailed view on the line card
show mpls forwarding prefix 10.0.0.4/32
show arp 10.0.13.3
show adjacency ten 0/2/0/1 detail location 0/2/cpu0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 49
3-Level Evenly Load BalancingBGP Multi-path, ECMP, Link Bundle
BGP multi-path z.z.z.za.a.a.a
1.1.1.1
1.1.2.2
1.1.8.8
11.1.1.1
11.x.x.32
BGP multi-paths
8-way (RLDI)
1.1.1.1
1.1.2.2
…
1.1.8.8
ECMP 32-way
(NR-LDI)
11.1.1.1
11.2.2.2
…
11.x.x.32
Link bundle, maximum 64
member ports (LAG)
Gig 0/0/0/1
Gig 0/1/0/2
…
Gig 0/3/0/39
z.z.z.z
Dynamic hashing table,
256 hashing buckets,
Evenly load balancing
regardless of odd or even
number of BGP
paths/ECMPs/bundle
member ports
Note, 8-way dynamic BGP multi-
path supported from 4.1.1 release
Level 1Level 2
Level 3
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 50
L3 Unicast – Load balancing Parameters
Incoming Traffic Type Parameters
IPv4 Unicast Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
IPv6 Unicast Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
MPLS
- IP Payload, with < 4 labels
Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
- IP Payload, with > 4 labels 4th MPLS Label (or Inner most) and Router ID
- Non-IP Payload Inner most MPLS Label and Router ID
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 51
L3 Multicast – Load balancing Parameters
Incoming Traffic Type Parameters
IPv4 Multicast (Bundle load-
balancing)
Egress LC selection Destination IP (a.k.a, multicast group address)
Egress NP selection Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
Egress NP selection Destination IP (a.k.a, multicast group address) (From 4.1 release for optimization)
Egress member selection
within an NP
Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
IPv6 Multicast
Egress LC selection Destination IP (a.k.a, multicast group address)
Egress NP selection Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
Egress NP selection Destination IP (a.k.a, multicast group address) (From 4.1 release for optimization)
Egress member selection
within an NP
Source IP, Destination IP, Source port (TCP/UDP only), Destination port (TCP/UDP
only), Router ID
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 52
L2 Services – Load balancing Parameters
Incoming Traffic Type Parameters
Bundle load-balancing
Default Source MAC, Destination MAC
EFP-based with auto mode XID of the xconnect
EFP-based with user hash User provided hash
Flow-based with IP src-dst Source IP and Destination IP
Flow-based with MAC src-dst Source MAC and Destination MAC
PW load-balancing over
ECMP
Default PW VC Label
Flow-based with IP src-dst Source IP and Destination IP
Flow-based with MAC src-dst Source MAC and Destination MAC
Note: Both Unicast and flood traffic follows the same load-balancing
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 53
Cisco ASR 9000 L2It does FULL L2, but not a classic switch
ASR 9000 support full L2 service (multi-point, point-to-point, E-Tree, native L2 or L2 over MPLS). It‟s fully MEF certified.
However, the behavior and configuration is quite different than traditional L2 switch
By default, it‟s Router, it doesn‟t do any bridging function unless you configure it specifically
There is no “switchport “ concept. All L2 features are done with “EVC” SW infrastructure
There is no global VLAN concept. All VLAN is local port significant. By default VLAN 10 on port A doesn‟t talk to VLAN 10 on port B. It require “bridge-domain” configuration to bridge port/VLAN together
ASR 9000 doesn‟t support SVI. No “int vlan” concept. Instead, it use “IRB/BVI”. It‟s different CLI but provide the same function
By default, STP is not enabled on any L2 port unless you “explicitly” configure it under STP process
NO global VLAN (no 4K limit)
Per-VLAN features, more flexible, more secure, more scale
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 54
ASR 9000 L2 “At-a-Glance”Built on New Cisco EVC SW Foundation
Flexible Cisco “EVC” Carrier Ethernet SW infrastructure
VLAN tag matching, manipulation, service
mapping
Distributed control plan High Multi-
dimensional L2 service scale, VLAN, PW,
Bridge-domain, VPLS, MAC address scale, etc
Distributed HW based MAC learning,
aging/flushing, 4-5Mpps
Rich L2 fast convergence
features
Standard compliant, MEF certified,
802.1q/qinq/.1ad/.1ah EoMPLS, H-VPLS/VPLS supported, L2 feature rich
E-OAM: CFM, Y.1731, Link OAM
Cisco “EVC”
Carrier Ethernet
Foundation
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 55
ASR 9000 MAC Learning Overview
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPUNP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA
CPU
Switch Fabric
MAC learning/aging/flushing is done by hardware
and fully distributed on each NP independently
Distributed HW based MAC Learning
–MAC learning /aging/flushing is fully distributed to the NP, independently
–Each NP can learn around 4-5 Millions MAC addresses per second in hardware
Data-plane MAC table synchronization
–MAC address are synchronized across all NPs in the entire system, regardless
of the bridge-domain or bridge port configuration.
–MAC address synchronization is achieved by special MAC notification
messages which is implemented in the data-plane microcode
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 56
Displaying MAC Table –How to handle large scale MAC address table
The Hardware based MAC tables must be collected in order to be displayed
From release 3.9, display of mac-address table requires a command to sync (collect) the Hardware MAC tables first. In prior releases, resync was constantly going as background task.
From 4.1, we have small enhancement: the system will check the total MAC address at first. If it has less than 16K, then it will display the MAC without any resync CLI. System will resyncimplicitly. However, if it has more than 16K MAC address, then it will behavior the same as in 3.9 release.
The command to display mac-addresses tells you the last resync time (hold „old‟ the displayed mac table is), and tells you the command to resync…
show l2vpn forwarding bridge-domain mac-address loc 0/7/CPU0To Resynchronize MAC table from the Network Processors, use the command...
l2vpn resynchronize forwarding mac-address-table location
Mac Address Type Learned from/Filtered on LC learned ResyncAge Mapped to--------------------------------------------------------------------------------0000.5e00.0101 dynamic Te0/7/0/1.100 0/7/CPU0 0d 0h 34m 40s N/A
After issuing the resync command, an info message will indicate when sync is complete.
l2vpn resyn forwarding mac loc 0/7/CPU0
l2fib[189]: %L2-L2FIB-5-MAC_TABLE_RESYNC_COMPLETE : The resynchronization of
the MAC address table is complete 0/7/CPU0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 57
Useful Show CLI – L2 Service
show run l2vpn
show l2vpn xconnect
show l2vpn xconnect group group1 xc-name vpws-1 detail
show l2vpn bridge-domain
show l2vpn bridge-domain bd-name vpls-1 detail
show l2vpn forwarding bridge-domain group1:132-vpls location 0/0/cpu0
show l2vpn forwarding bridge-domain group1:132-vpls detail location 0/0/cpu0
show l2vpn forwarding interface gig 0/0/0/15 detail location 0/0/cpu0
show controllers np struct 18 all loc 0/0/cpu0 NP structure 18 is for MAC address
show controllers np struct 18 detail all loc 0/0/cpu0
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 58
“8+1” CPU for Ultra-High multi-
dimensional control plane scale
ASR 9000 Scalable System Architecture
FIC
CPU BITS/DTI
Line Card
FIA
CPU
RSP
Control plane split among RSP and LC
CPU (same type of CPU as RSP)
L2 protocols, BFD, CFM, Netflow runs
on the LC CPU for high scale
Multi-dimensional scale
FIB 1.3M (v6 use two entries)
MAC address 512K
L2 interfaces 64K
Pseudowires 64K
Bridge-domain/VFI 8K
EFPs per BD 16K
L3 interfaces, VRF 8K
CFM MEP/MIP 32K/LC
Netflow 1M entries/LC, 100K pps/LC
BFD 1250/LC (with 150msechello timer), minimal hello
timer is 15msec
Scale examples
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 59
Scale Profiles
RP/0/RSP0/CPU0:asr1(admin-config)#hw-module pro scale ?
default Default scale profile
l3 L3 scale profile
l3xl L3 XL scale profile
RP/0/RSP0/CPU0:asr1(admin-config)#hw-module pro scale l3
Mon Apr 4 18:55:40.070 UTC
In order to activate this new memory resource profile, you must manually reboot the
line cards.
RP/0/RSP0/CPU0:asr1(admin-config)#hw-module pro scale l3xl
Mon Apr 4 18:55:52.654 UTC
In order to activate this new memory resource profile, you must manually reboot the
system.
Feature Default L3 L3XL
FIB
(v4 and v6 share the
same table, v6 use
two entries)
512K v4 1M v4 1.3M v4
256K v6 (128K
maximum global and
per-VRF)
512K v6 (128K
maximum global and
per-VRF)
650K v6 (256K
maximum global and
per-VRF)
MAC 512K 128K 32K
Bridge-domain/VFI 8K 8K 2K
Note, FIB scale is different than the RIB. FIB is hardware table in the forwarding ASIC. RIB is software table on the
RP. ASR9K can support more than 20M RIB
3 scale profiles
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 60
Access
Ethernet Node
Ethernet Node
DSL Node
PON Node
HA and Fast Convergence
Access
Distribution Node
Pre-Aggregation/aggregation node
IP/Service Edge
Business
Corporate
Business
Corporate
2G/3G RBS
Residential
STB
Residential
STB
Residential
STB
Business
Corporate
Cell Site Gateway
IP/MPLS
Residential Data/Voice
Business L3 VPN
Residential IPTV/VoD
Business L2 VPN (E-Line/E-LAN/E-Tree)
Mobile backhaul
L2 and L3 transport layer resiliency, transit link or transit node fail, L2 or L3 transport
protocol re-converge. Service layer is not aware
L2: MST, G.8032, REP(not supported), PVST(not supported)
L3: IGP/BGP, TE/FRR(link, node, path protection), IP/FRR (ISIS per-link and per-prefix
, OSPF per-link, OSPF per-prefix IP/FRR
Multicast: PIM, P2MP-TE/FRR
Link layer resiliency: transport and service layer is not aware, for link protection only Link Bundle (active-active, 1:1 backup, m:n backup)
L2/L3 AccessIP/MPLS (pre)aggregation
L2/L3 Link Bundle
System HA: IOS-XR Modular OS, Micro-kernel, RSP failover, SMU, ISSU(4.2), Process restart. All protocols are SSO, all protocol support NSF/GR, OSPF/ISIS/LDP/BGP support NSR. PIM support NSF and is NSR compliant
Resiliency infrastructure: distributed BFD, distributed EFD, HW based LoS detection, IPoDWDM FEC/EFC & proactive protection, Hierarchical Forwarding table structure and in-place modification, Fast programming FIB table
Service layer resiliency, service termination point (node or link) fail., service is down
and need to re-route
L2: PW redundancy, L3: BGP PIC, Multicast: MoFRR
Access dual home: REP-AG/MST-AG, Rapid-PVST/PVST-AG, MC-LAG, G.8032,
Flexible Link (not supported), HSRP/VRRP
State sync between primary and backup service node: ANCP, IGMP snooping,
DHCP/ARP/IGMP (in future), etc
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 61
FIAFIA
RSP1
Switch Fabric Bandwidth Access OverviewIntelligent Fabric and Internal System QoS
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
Crossbar
Fabric
ASIC
RSP0
1: Fabric Request
3: Fabric Grant
2: Arbitration
4: load-balanced transmission across fabric links
5: credit return
Ingress LC Egress LC
Arbiter
Arbiter
3 strict priority scheduling/queueing
Back pressure and virtual output queue
Multicast and Unicast separation
(separated queues and fabric plane)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 62
End-to-End System QoS Queuing
Ingress (sub-)interface QoS Queues
VirtualOutputQueues
Egress FIAQueues
End-to-End priority (P1,P2, Best-effort) propagation Guarantee bandwidth, low latency for high priority traffic
at any congestion point3 strict priority level across all internal HW components
Configure withIngress MQC
4-layer hierarchyTwo strict high priority +
Normal priority
Egress (sub-)interface QoS Queues
Configure withEgress MQC
4-layer hierarchyTwo strict high priority +
Normal priority
Implicit ConfigurationTwo strict high priority +
Normal priority
Ingress side of LC Egress side of LC
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPUNP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA
CPU
Switch Fabric1
23
4
12
34
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 63
Backpressure and VoQ Mechanism
Egress NP congestion backpressure to ingress FIA
Packet is en-queued in the dedicated VoQ
No impact of the packet going to different egress NP
No head-of-line-block issue
Ingress side of LC1 Egress side of LC2
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPUNP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA
CPU 1
3
2
Backpressure: egress NP egress FIA fabric Arbiter ingress FIA VoQ
Switch Fabric
VoQ Scale: Each FIA has P1/P2/BE queue set for every NP and RSPs in the entire system
5Gbps
10Gbps
5Gbps
Packet going to different egress NP put into different VoQ set
Congestion on one NP won‟t block the packet going to different NP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 64
CLI – System Queues Counters
RP/0/RSP0/CPU0:PE2-asr#show controllers fabric crossbar ? switch fabric
readreg Read register
serdes Display chico info
statistics Packet Statistics switch fabric drop counters
RP/0/RSP0/CPU0:PE2-asr#show controllers fabric fia ? fabric interface ASIC
bridge show command for bridge asic
drops Drops for fabric interface asic drop counters
errors Errors for fabric interface asic
link-status Link status between fabric/xbar and fabric interface asic
q-depth Ingress Queue Depth Counters
stats Packet statistics for fabric interface asic packet statistics
trace Show controllers fia trace data
RP/0/RSP0/CPU0:PE2-asr#show controllers fabric fia bridge ? bridge FPGA
ddr-status DDR staus between bridge and fia
flow-control Flow control information for bridge flow control counters
stats Packet statistics for bridge packet statistics
sync-status Bridge and NP sync status
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 65
Port QoS Overview
High scale
–Up to 3 Million queues per system (with E linecard)
–Up to 2 Million policers per system (with E linecard
Highly flexible: 4 layer hierarchy queuing/scheduling support
–Four layer scheduling hierarchy Port, Subscriber Group, Subscriber, Class
–Egress & Ingress, shaping and policing
Three strict priority scheduling with priority propagation
Flexible & granular classification, and marking
–Full Layer 2, Full Layer 3/4 IPv4, IPv6
NP0 PHY
NP2 PHY
NP3 PHY
NP1 PHYFIA
CPUUser configure port level QoS via MQC
CLIs
QoS function is done by special Queuing
HW chip within the NP complex
Different memory options have different
scale
Q
Q
Q
Q
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 66
Agenda
Hardware Architecture
‒ Chassis, RSP, Line Cards and Switch Fabric
Software Architecture
‒ Packet Forwarding Overview
‒ L3 Forwarding
‒ L2 Forwarding
‒ Load Balancing (BGP Multi-path, ECMP, Link Bundle)
‒ HA and Fast Convergence
‒ QoS Architecture
Data Plane Troubleshooting
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 67
Switch Fabric
NP0PHY
NP2PHY
NP3PHY
NP1PHY
FIA
CPU
B0
B1
show interface gig 0/0/0/35show|clear controllers gig 0/0/0/35 stats
show|clear control np counters np0 location 0/0/cpu0
clear control fabric fia location 0/0/cpu0show controllers fabric fia bridge stats location 0/0/cpu0
clear control fabric fia location 0/0/cpu0show controllers fabric fia stats location 0/0/cpu0
Gig0/0/0/35
Gig0/0/0/0
Ingress
Egress
Show interface gig 0/0/0/0show|clear controllers gig 0/0/0/0 stats
show|clear control np counters np3 location 0/0/cpu0
clear control fabric fia location 0/0/cpu0show controllers fabric fia bridge stats location 0/0/cpu0
clear control fabric fia location 0/0/cpu0show controllers fabric fia stats location 0/0/cpu0
Show controller fabric crossbar stats instance [0-1] location 0/x/cpu0
Data Plane CountersFocus on the PHY/NP counters,
NOT B/FIA/crossbar
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 68
Interface
“show interface Ten 0/1/0/3 “
Displays interface related statistics
“show controller Ten 0/1/0/3 stat”
Find out if there are any drops or errors at the physical layer
Input drop overrun = 0
Input drop abort = 0
Input drop invalid VLAN = 0
Input drop invalid DMAC = 0
Input drop invalid encap = 0
Input drop other = 0
Input error giant = 0
Input error runt = 0
Input error jabbers = 0
Input error fragments = 0
Input error CRC = 0
Input error collisions = 0
Input error symbol = 0
Input error other = 0
RP/0/RSP0/CPU0:A9K-TOP#sh int te 0/3/0/3 | i opsMon Mar 21 14:22:43.831 UTC
304292 packets input, 36389589 bytes, 47401 total input drops
4514 drops for unrecognized upper-level protocol107885 packets output, 10123636 bytes, 0 total output drops
RP/0/RSP0/CPU0:A9K-TOP#sh int te 0/3/0/3 | i err
Mon Mar 21 14:23:00.856 UTC0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 output errors, 0 underruns, 0 applique, 0 resets
Aggregate drops from mac/phy and np
Phy errors (controller related)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 69
Which interface to which NP?
Use the command “show controller np ports all location 0/Y/CPU0” to find out which interface is connecting to which NP
RP/0/RSP1/CPU0:A9K-BOTTOM#sh controllers np ports all loc 0/7/CPU0
Fri Mar 4 12:09:41.132 EST
Node: 0/7/CPU0:
----------------------------------------------------------------
NP Bridge Fia Ports
-- ------ --- ---------------------------------------------------
0 0 0 TenGigE0/7/0/3
1 0 0 TenGigE0/7/0/2
2 1 0 TenGigE0/7/0/1
3 1 0 TenGigE0/7/0/0
In case you have an oversubscribed linecard, 2 TenGig interfaces can be connected to the same NP.
For 80G linecards such as the A9K-8T, you'll see 8 NP's listed.
For the 40 port 1GE, you'll see 10 1-Gig interfaces connecting to an NP, note that this is not oversubscription.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 70
NP Counters (not all are drop or error related)
RP/0/RSP1/CPU0:A9K-BOTTOM#show controller np count np1 loc 0/7/CPU0Node: 0/7/CPU0:
----------------------------------------------------------------
Show global stats counters for NP1, revision v3
Read 30 non-zero NP counters:
Offset Counter FrameValue Rate (pps)
-------------------------------------------------------------------------------
22 PARSE_ENET_RECEIVE_CNT 51047 1
23 PARSE_FABRIC_RECEIVE_CNT 35826 0
30 MODIFY_ENET_TRANSMIT_CNT 36677 0
31 PARSE_INGRESS_DROP_CNT 1 0
34 RESOLVE_EGRESS_DROP_CNT 628 0
40 PARSE_INGRESS_PUNT_CNT 3015 0
41 PARSE_EGRESS_PUNT_CNT 222 0
Received from the ethernet framer/mac/phy
Received from fabric
Packets modified and sent to ethernet
Packets dropped in the PARSE stage
Packets received and determined to be punted to the CPU.
NP Drop counters on CCO: https://supportforums.cisco.com/docs/DOC-15552
Drops for unrecognized upper-level protocol error: https://supportforums.cisco.com/docs/DOC-15490
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 71
Cisco ASR 9000 Summary
Most Scalable, Lowest-Power
Carrier Ethernet System in the
Industry
Best in Class L2 + L3, Carrier Ethernet, &
MSE Services
Optimized for Video Services over Converged
Wireline + Wireless NGNs
Nonstop Availability with IOS-XR Modular
OS
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco ConfidentialPresentation_ID 72
Thank you.