Cisco CRS-3 Carrier Routing System MultiChassis Overviewd2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKARC-3002.pdf · Cisco CRS-3 Carrier Routing System MultiChassis Overview Session

© 2012 Cisco and/or its affiliates. All rights reserved. BRKARC-3002 Cisco Public

Cisco CRS-3 Carrier Routing System

MultiChassis Overview Session ID – BRKARC-3002


Agenda

CRS (Carrier Routing System) Overview

CRS-3 Overview

Overview of CRS Data Path

Overview of CRS Switch Fabric

CRS MultiChassis Specifics

CRS MultiChassis Switch Fabric details

CRS MultiChassis Control Ethernet

CRS MultiChassis Configuration

CRS MultiChassis Troubleshooting (Control Ethernet & Fabric)

3


Agenda – continued Appendix for reference

Appendix Typical migration Step SC to MC

Appendix A Case Study 1 - Offline SC to MC migration

Appendix B Case Study 2 – Online SC to MC migration

Appendix C FCC Physical Installation notes

4


CRS (Carrier Routing System) Overview

A fully modular and distributed routing system

CRS-1 (40G) and CRS-3 (140G) Systems

4,8 & 16 Slot Standalone Systems

8 & 16 Slot MultiChassis Systems (LCC‟s + FCC‟s)

5


CRS-1 & 3 Hardware Introduction

16 slot Line Card

Chassis with

integrated rack

(standalone or LCC)

External

Interfaces (PLIM)

Redundant

Route Processors

Fabric Card

Chassis (FCC)

8 slot Line Card

Chassis

rack mountable

Front and Rear

Access Required

MSC and Fabric

Access - Rear

6


CRS-3 Overview

New 140G Fabric and Line Cards

Increased throughput and scale

Backwards compatible with CRS-1 Hardware

4,8 & 16 Slot Standalone Systems

8 & 16 Slot MultiChassis Systems (LCC‟s + FCC‟s)

Can leverage 40G Line Card's & MSC‟s

7


CRS-3 Product Family

Foundation for the IP NGN Core

1.12 Tbps 2.24 Tbps 4.48 Tbps 4.48 Tbps to 322 Tbps

CRS-4/S CRS-8/S

CRS-16/S CRS-MC

Unprecedented service flexibility

Continuous system operation True Telco grade OS—IOS-XR

In-service hardware and software upgrades

Hitless upgrade to Multichassis

Unparalleled system longevity Multi-chassis fabric scales to 322Tbps.

Investment protection—common forwarding engines and I/O modules Modular hardware and software

8


CRS-3 Line Card Architecture

High Level LC Architecture

MSC Architecture

‒PSEs (Ingress and Egress)

‒IngressQ

‒FabricQ

‒EgressQ

Switch Fabric Architecture

Control Ethernet

9


CRS-3 System Overview

MAC 100G PHY

PLA

• Same architecture as CRS-1 but at 140G

• Compatible with all CRS-1 chassis sizes

• Standalone or MultiChassis

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3

S3

PLIM MSC/FP

Fabric

PSE

EgressQ

PSE

Intel CPU Sub-system

FabricQ

FabricQ

IngressQ

1 of 8 2 of 8

3 of 8 4 of 8

5 of 8 6 of 8

7 of 8 8 of 8

S1/S2/S3

S2

10


CRS-3 Line Card

from PLIM

120G 2x80G

PSE

160G

160G EgressQ

PSE

100G

100G


141G

113G

113G FabricQ

FabricQ

IngressQ

Output Queuing Output Features Multicast Replication Cell Reassembly

Forwarding Lookup Input Features

Queuing for Fabric Cell Segmentation Input Shaping

to PLIM

to Fabric

from Fabric

160G 2x100G

11


CRS-3 New hardware

CRS-3 New Hardware Boards Supporting Platforms

CRS-4 S3 Fabric Single Chassis CRS-4



CRS-16 S13 Fabric MultiChassis Line Card Chassis

FCC S2 Fabric Board MutiChassis Fabric Card Chassis

CRS-3 MSC 140G All CRS-3 platforms

CRS-3 14x10GE PLIM All CRS-3 platforms



Note: CRS-3 Fabric is backward compatible with CRS-1 40G MSC/PLIMs. Thus, a CRS-3 MultiChassis will support CRS-1 Line Card chassis. CRS-3 PLIMs are compatible only with CRS-3 MSCs and vice versa.

12


CRS-3 MSC140 vs. FP140

MSC140 FP140

Feature Without licenses With appropriate licenses

Queues 64K / slot 8 / port 8 / port

L3 Interfaces 12K / slot 250 / slot 250 / slot

Multichassis Yes No Yes

Netflow Sampling <1: 1500 >1: 1500 <1:1500

L2/L3VPN 2K+ VRFs/LC No VRFs 250 VRFs + L2VPN connections per LC

Route Scale 4M IPv4/2M IPv6 1M IPv4/500K IPv6 4M IPv4/2M IPv6

TE Scale >3K midpoints 3K midpoints + heads + tails >3K midpoints

Advanced Features Tunneling, LI No Tunneling or LI Tunneling, LI

13


CRS System Attributes Comparison MSC40 CRS-1 MSC140 CRS-3

Bandwidth 40 Gbps 140 Gbps

Max Packet per second 80 Mpps 125 Mpps

FIB Scalability 2M IPV4, 1M IPV6 4M IPV4, 2M IPV6

BW Modes 20/40 Gbps 40/140 Gbps

Queues/Groups/Ports 8k/2k/768 64k/16k/128

Supported Fabric Cards 40G and 140G fabric 140G fabric only

PLIMs

8 x 10GE

16xOC48, 4xOC192, 1xOC768

Modular 6 x SPA

14 x 10GE

20 x 10GE (oversubscribed)

1 x 100GE

SONET and Modular PLIMs (future)

Power

LC (MSC + PLIM): 530W

Switch Card (4/8/16 slot): 102/185/206W

Total (4/8/16-slot): 3.5/6.6/11.5KW

LC (MSC + PLIM): 600W

Switch Card (4/18/16 slot): 49/90/94W

Total (4/8/16 slot): 4/7.5/14 KW

14


Line Card Chassis (LCC) Fabric Card Chassis (FCC)

CRS-3 Full Rack MultiChassis

100m FRONT: 24 Fabric cards

2 Shelf Controllers

Control Ethernet Connections

BACK: 24 Optical Interconnect Modules (OIM)

2 OIM LED Module

Array Cable Connections

•Optical Backplane

•Redundant Fans/Power

•Supports CRS-16

FRONT: 16 Interface Slots

2 RP Slots

2 Controller slots BACK:

16 LC Slots

8 Fabric Card Slots

•Mid-Plane Design

•Redundant Fans/Power

•140G per Slot

• Each Line Card Shelf add 4.48 Tbps to the MultiChassis system

• Fabric architecture can support 72 LC Chassis and 8 Fabric Chassis - 322 Tbps

• Hitless Single Chassis to MultiChassis Upgrade 15


Overview of CRS Data Path

High Level view Switch Fabric and Fabric attributes

Basic Fabric Building Blocks

CRS-1 and CRS-3 Fabric Bandwidth Comparison

High Level Data Path Ingress to Egress via Fabric

16


Switch Fabric:

• Three Stage (S1, S2, S3) Benes Topology

• Multicast replication in Fabric

• 2 levels of priority through Fabric

HP Low latency path LP Best effort traffic

1 of 8

2 of 8

8 of 8

1

2

8

1

2

16 Line Card

MSC Line Card

MSC

S1 S3

S1 S3

S1 S3 S2

S2

S2

Cisco CRS Switch Fabric High Level

Ingress Side: Each Line Card (and RP) is connected to all Fabric planes through the S1 stage

Line cards chop Packets up into “Cisco Cells” for transport across the fabric

Destination line card (or RP) is prepended to the Cell

Egress Side: Each Line Card (and RP) is connected to all Fabric planes through S3 stage

Line cards reassemble Packets from “Cisco Cells” for egress packet processing

17


Cisco CRS Switch Fabric Attributes

Redundancy

• Fabric implemented across 8 fabric planes

• Loss of any one fabric plane does not reduce capacity

• Loss of additional planes reduces capacity by 1/7 per plane

Hardware Capacity:

• Fabric implements at 1296 x 1296 buffered non-blocking switch

• Provides Capacity for 72 LCC

72 LCC * 16 Line cards = 1152

72 LCC * 2 Route Processors = 144

Multicast

• Full multicast capability in the fabric

• 1M fabric mcast groups

• Mcast cells are dropped if fabric is congested

• Mcast cells queued separately from unicast

1 of 8

2 of 8

8 of 8

1

2

8

1 2

16 Line Card

MSC

Line Card

MSC

S1 S3

S1 S3

S1 S3 S2

S2

S2

Queuing:

• Discrete Queues for control and data plane

• Integrated Congestion & flow control

18


CRS-3 Basic Fabric Building Blocks

160G

100G

100G

141G

113G

113G

IngressQ

FabricQ

FabricQ

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3

S3

1 of 8 2 of 8

3 of 8 4 of 8

5 of 8 6 of 8

7 of 8 8 of 8

IngressQ segments packets into cells

FabricQ reassembles packets from cells

Fabric routes cells to egress card

S2

19


CRS Fabric Bandwidth Comparison

IngressQ

FabricQ

S1

S3

2.5G x 4 = 10Gbps ingress (per plane) 10G * 8 planes = 80Gbps total ingress

FabricQ

S3

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

2.5G

IngressQ

FabricQ

S1

S3

FabricQ

S3

5G

5G

5G

5G

5G

2.5G * 8 = 20Gbps egress (per plane) 20G * 8 planes = 160Gbps total egress

5G * 8 = 40Gbps egress (per plane) 40G * 8 planes = 320Gbps total egress

5G

5G

5G

5G

5G

5G

5G

5G

MSC40 MSC140 Fabric Fabric

CRS-1 CRS-3 5G x 5 = 25Gbps ingress (per plane) 25G * 8 planes = 200Gbps total ingress

20


Fabric Bandwidth Calculations

CRS-3 • IngressQ to S1

‒ = 200Gbps (raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 141Gbps

• S3 to FabricQ

‒ = 320Gbps (raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 225Gbps

or 113Gbps per FabricQ

CRS-1 • IngressQ to S1

‒ = 80Gbps(raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 56Gbps

• S3 to FabricQ

‒ = 160Gbps(raw bw) * 8b/10b (encoding) * 120/136 (cell overhead) = 112Gbps or

56 Gbps per FabricQ

21


CRS Fabric Bandwidths and Links Between

MSC40 and MSC140 MSC40 MSC140

To Fabric

Ingress

IngressQ has 32 links @2.5G for 40G

= 32 links * 2.5Gbps *8/10 (coding) *120/136 (cell tax) = 56Gbps

(This is for 8 planes)

Scaling IngressQ to 140G requires 40 links @5G

IngressQ to Fabric ASIC S1

= 40 links * 5Gbps *8/10 (coding) *120/136 (cell tax) = 141Gbps

(This is for 8 planes)

From Fabric

Egress

2 FabricQ ASICs per MSC, each with 2 RX Links per S3 ASIC. 64 links @ 2.5Gb

= 64 links * 2.5Gbps *8/10 (coding) *120/136(cell tax) = 100Gb

(this is for 8 planes)

100 Gb is split across 2 FabricQ ASICs on the LC. Egress forwarding capacity

is only 40Gb, so the LC must backpressure when overloaded

Fabric ASIC S3 to FabricQ

= 64 links * 5Gbps * 8/10 (coding) *120/136 (cell tax) = 225Gbps or 113Gbps per FabricQ.

(for 8 planes)

22


IngressQ

To-Fabric interface

Every MSC and RP transmits data to the fabric via an IngressQ ASIC

3072 fabric destinations with 2 priorities, plus 2 multicast queues

An IngressQ ASIC has 48 TX links @5Gbps

‒ 40 used for 200Gbps raw bandwidth

‒ 141 Gbps net bandwidth

TX links physically connect to each of the fabric planes

MAC 100G PHY

PLA

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3

S3

PSE

EgressQ

PSE


FabricQ

FabricQ

IngressQ 1 of 8 2 of 8

3 of 8 4 of 8

5 of 8 6 of 8

7 of 8 8 of 8

S2

23


S1 Plane 1

Ingr

ess

LC

•With 8 fabric planes and 6 connections to the S1 ASICs, each CRS-3 LC has 48 connections. 40 of these links are

used for carrying data traffic.

•40 connections @ 5Gb = 200Gb

•8b/10b encoding reduces this to 160Gb effective BW

•Cell tax means that the final result is ~140Gb

S1 Plane 3

S1 Plane 5

S1 Plane 7

S1 Plane 2

S1 Plane 4

S1 Plane 6

S1 Plane 8

IngressQ to S1

24


S1/S2/S3 ASIC‟s

Fabric switching ASIC

5Gbps SerDes with 2.5Gbps backwards compatibility mode

HP/LP unicast and HP/LP multicast queues

MAC 100G PHY

PLA

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

PSE

EgressQ

PSE


FabricQ

FabricQ

IngressQ 1 of 8 2 of 8

3 of 8 4 of 8

5 of 8 6 of 8

7 of 8 8 of 8

25


S1-> S2 -> S3 switching

Ingress linecard selects which FabricQ ASIC on the egress linecard a packet should be sent to.

The selection is encoded in the Cell Header when the packet is converted into Cisco Cells

S1 switches cells streams, load-balancing across the three S2 ASICs

S2 queues and routes the cell to the correct S3 based on the Cell header

S3 queues and routes the cell to the correct FabricQ based on the Cell header

72 links from S1 stage to the 3 S2 ASICs (24 links to each ASIC)

144 links from S2 stage to the 2 S3 ASICs (24 links to each ASIC)

S1

S1 S3

S3

S2

S2

S2

26


Switch Fabric Multicast Replication

CRS provides efficient multicast replication via 3 operations

S2 can replicate cells to registered (via FGID) S3 SEAs

S3 can replicate cells to registered (via FGID) FabricQs

Egress SPP can replicate packets for each output port

1 million Fabric Group IDs

Program fabric for mcast

S2 and S3 lookup FGID

S2 S3 S1

S1 switches to all S2s

Mcast replication at S2 based on FGID

Replication at S3 based on FGID

27


Egress CRS-3

Fabric Plane

•On egress, in the CRS-3 LCC chassis,

there are 2 S3 ASICs per fabric plane.

(the same S13 ASIC acts as S1 or S3

for different Fabric stages

•In the 16-slot chassis, each S3

receives 72 out of 144 TX links in

service.

•Each MSC receives 8 TX links per S3

„pair‟

•In the 16-slot chassis, each RP

receives 4 TX links per S3 pair

S1

S1 S3

S3 S2

S2

S2

28


FabricQ CRS-3

From-Fabric interface

Every MSC and RP receives data from the fabric via one or more FabricQ

ASICs

‒ MSCs have two FabricQ ASICs

‒ RPs have one FabricQ ASIC

A FabricQ ASIC has 32 RX links @5Gbps

‒ 160 Gbps raw, 113 Gbps net bandwidth

‒ MSC140/FP140 uses 2 FabricQs for 225 Gbps total

RX links physically connect to each of the fabric planes

MAC 100G PHY

PLA (Beluga)

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3 S3

S3 S3

S1

S1

S2

S2

S3

S3

PSE (Pogo)

EgressQ (Tor)

PSE (Pogo)


FabricQ (Crab)

FabricQ (Crab)

IngressQ (Seal)

1 of 8 2 of 8

3 of 8 4 of 8

5 of 8 6 of 8

7 of 8 8 of 8

S2

29


S3

S3 Plane 1

Egre

ss L

C

S3

S3 Plane 2

S3

S3 Plane 3

S3

S3 Plane 4

S3

S3 Plane 5

S3

S3 Plane 6

S3

S3 Plane 7

S3

S3 Plane 8

•In the 16-slot chassis, there are 2*S3 ASICs per plane connected to each MSC/RP

•2 FabricQ ASICs per MSC, each with 4 RX Links per S3 ASIC

•8 connections per linecard per fabric plane

•64 connections in total @ 5Gb = 320Gb

•8B/10B encoding reduces BW to 256Gb

•cell tax reduces this down to 225Gb

•225Gb is split across 2 FabricQ ASICs on the LC

S3 to FabricQ CRS-3

30


Overview of CRS Switch Fabric

Fabric stages defined

Fabric Planes

Fabric Resilience

31


Cisco CRS-1 Switch Fabric Overview High Speed Path for Transit Packets and IPC

1 of 8

2 of 8

8 of 8

1

2

8

40 Gbps 100 Gbps

Line Card Line Card

8 independent fabric planes 2.5x speedup through fabric Support for 72 chassis & 1296 RP/MSC clients

32


Cisco CRS-1 Switch Fabric Overview

3 Stages with Priority

1 of 8

2 of 8

8 of 8

1

2

8

1

2

16

Line Card Line Card

S1 S2 S3

S1 S2 S3

S1 S2 S3

3 stage switching fabric

High and low priority cells Set on control by default Set on transit pkts via CLI Vital Bit for IPC

33


CRS-3 Switch Fabric - Simplified Form

The 8 fabric planes providing ingress BW is 200Gb which reduces to ~140Gb effective capacity due 8/10B encoding and cell tax overhead

On egress, each linecard has 2 ports per plane at 20Gb. 40Gb per fabric plane * 8 planes = 320Gb. 8B/10B encoding reduces BW to 256Gb. Cell tax reduces down to 225Gb

Egress PSE ASIC forwarding capacity is ~145Gb, so the LC must backpressure when overloaded

34


Decisions at Each Stage in 16 slot CRS-3 For each of the 8 planes

S2 S3 S1

Send to any S2. No need to look at header

Look at cell header. Send to S3 based on chassis

Look at cell header. Send to specific LC and FabricQ

35


CRS-3 Fabric Planes

A full fabric consists of 8 planes

Switch Element ASICs (SEAs) perform switching operations

Each plane is an independent set of SEAs

A Switch Element ASIC can act as S1, S2 or S3 stages of Fabric. The S1 and S3 stages are in the same ASIC.

SEAs have 72 RX and 144 TX links

Number of RX and TX active depends on programming/stage

Cells never move between planes

Each plane will have multiple S13 (S1 and S3 stages) and S2 asics. E.g. It will have 2 S13 asics and 3 S2s. The S13 asic will act as both S1 and S3 stage on the LCC.

36


CRS Fabric Resilience and System Recovery

CRS can operate with missing/failed fabric components

System can disable individual ―link‖ but still use plane

BW Capacity reduced if links or entire planes are down

Full capacity for Intermix traffic can be reached with 7 planes

At least 2 planes must be active for the system to work

Specifically one odd and one even numbered plane must be up.

Each cell is sent on a single plane

Cells have error correcting ECC, but

No multi-plane reconstruction of errored cells

Removing fabric HW w/o shutting down the plane first will lose cells

37


CRS MultiChassis Specifics

SC vs. MC (Single Chassis vs. MultiChassis)

Fabric details

MultiChassis benefits

MC building blocks

Fabric Plane config, cabling, and SFC placement

Sample MC Systems

CRS System Failure and Recovery

38


CRS Single Chassis vs. MultiChassis

In MultiChassis, FCC provides the Fabric switch connectivity between

different Line Card Chassis which house the RPs, MSCs, PLIMs and

DRPs.

The Control Ethernet network is used for processes on different devices to

communicate for functions such as system device discovery, image

transfers, heartbeat messages, alarms and configuration management

Spanning Tree Protocol is used between the Switch elements on the Shelf

controller and on the inter-RP links in order to prevent loops

39


CRS MultiChassis and Benes Fabric

The distinct stages of the Benes fabric allow easy migration to MultiChassis

‒Line cards (MSC) and S1 and S3 stages stay on LCC

‒S2 Stage move to the FCC

‒Array Cables connect the fabric stages together

Upgrading from single chassis to MultiChassis can be done in-service

‒The S2 stage for each plane is moved to FCC one at a time

‒LCC requires new fabric cards with just the S1 and S3 stages

MSC

S1 S3

S1 S3

S2

S2

MSC

Up to 100m

Up to 100m

Line Card Chassis (LCC)

Line Card Chassis (LCC)

Fabric Card Chassis (FCC)

40


CRS Fabric Chassis (FCC)

Front & rear access – mini-midplane for control network access

Front

‒24 Fabric card

‒2 Shelf Controllers

Back

‒24 OIM slots

‒2 Fiber LED modules

Dimensions:

‒23.6‖ W x 41*‖ D x 84‖ H

‒(60 W x 104.2 D x 213.36H (cm))

Power: ~9 KW DC, 11.1 KW AC

Weight: ~1550 lbs/704kg

Heat Dis.: 27600 BTUs

41


What benefits to expect from a CRS

MultiChassis System Scale

Process Placement to distribute the process load

Redundancy and Reliability

Performance

42


MultiChassis technology building blocks

Key points:

‒A fabric plane can span multiple S2 fabric boards

‒Multiple fabric planes can be supported in a Fabric chassis

1 0 2 3

4 5 6 7

1 0 2 3 5 4 6 7 0 1 2 3

4 5 6 7

1 FCC

2 FCCs

4 FCCs

43


MultiChassis technology - building blocks

0

8 FCCs

1 2 3

4 5 6 7

44


Fabric Plane Configuration

Multi-Module Topology

Three Plane Configuration

Three S2 SFC‘s are used to create a plane

‒The array cables for each fabric plane are connected to all three cards

Each S13 card is connected to 3 different S2 cards

To LC rack # 0

To LC rack # 2

To LC rack # 8

To LC rack # 6

Single Module Topology

Full Plane Configuration

Each S2 SFC serves one plane The array cables for each fabric plane are connected to a

common SFC

Each S13 card is connected to a single S2 card

To LC rack # 1

To LC rack # 0

Switch Fabric Card (SFC) Switch Fabric Cards

45


MultiChassis Cabling

4+4 Configuration

‒4 CRS-16 Line Card Chassis

‒4 Fabric Card Chassis

2+1 Configuration 2 CRS-16 Line Card Chassis

1 Fabric Card Chassis

46


SFC – Switch Fabric Card Placement The CRS MutiChassis system supports the placement of all the OIMs and fabric

cards in a single chassis or across multiple chassis

‒CRS Fabric Chassis installation on cisco.com provides guidelines and best practices for fabric card placement

The CRS MultiChassis systems support configuration with one, two, or four Fabric Chassis

Additional chassis‘ add fabric redundancy, not additional capacity

‒Protects fabric bandwidth in case of the loss of single Fabric chassis

‒The CRS fabric only needs 7 planes to run at full capacity

‒The 8th plane provides active load sharing redundancy

‒The CRS fabric capacity degrades with losses of additional planes

‒ Loss of additional planes reduce throughput by approximately 1/7

The CRS fabric requires at least 1 odd plane and one even plane to function

MultiChassis capacity is gated by the OIM hardware

‒Up to 3 CRS-16 LCCs, in single topology mode where 8 OIM/SFC are required

‒Up to 9 CRS-16 LCCs, in multi-module topology mode where 24 OIM/SFC are required

47


OIM LED Module OIM LED Module provided to aid Operations

‒Two OIM LED Modules per Fabric Chassis

‒LEDs provide visual indication of the status of each array cable

‒Each tri-color LED shows 5 states:

OK (green)

no signal (none)

misconnected (blinking red)

signal fault (yellow)

connect here (slow blinking green)

OIM LED Module provides an installation aid to help identify misconnected fiber bundles

‒Based on fabric configuration, a point-point connection map will be generated by the router

‒If only 1 fiber bundle is misconnected, specifies the correct place to connect or reconnect a fabric cable.

OIM LED Module

48


Fabric Placement with Two CRS-16 LCC

Single Topology Example

SFC 0

SFC 1

SFC 2

SFC 3

SFC 4

SFC 5

SFC 6

SFC 7

SFC 8

SFC 9

SFC 1

0

SFC 1

1

SFC 1

2

SFC 1

3

SFC 1

4

SFC 1

5

SFC 1

6

SFC 1

7

SFC 1

8

SFC 1

9

SFC 2

0

SFC 2

1

SFC 2

2

SFC 2

3

She

lf Co

ntro

ller 0

Sh

elf C

on

trolle

r 1

Full Rack FCC Front of Chassis

Full Rack FCC Back of Chassis

OIM

LED

1

OIM

LED

0

2+2 FCC Single Topology

SFC 0

SFC 1

SFC 2

SFC 3

SFC 4

SFC 5

SFC 6

SFC 7

SFC 8

SFC 9

SFC 1

0

SFC 1

1

SFC 1

2

SFC 1

3

SFC 1

4

SFC 1

5

SFC 1

6

SFC 1

7

SFC 1

8

SFC 1

9

SFC 2

0

SFC 2

1

SFC 2

2

SFC 2

3

She

lf Co

ntro

ller 0

Sh

elf C

on

trolle

r 1



OIM

LED

1

OIM

LED

0

Legend :

SFC 0

SFC 1

SFC 2

SFC 3

SFC 4

SFC 5

SFC 6

SFC 7

SFC 8

SFC 9

SFC 1

0

SFC 1

1

SFC 1

2

SFC 1

3

SFC 1

4

SFC 1

5

SFC 1

6

SFC 1

7

SFC 1

8

SFC 1

9

SFC 2

0

SFC 2

1

SFC 2

2

SFC 2

3

She

lf Co

ntro

ller 0

Sh

elf C

on

trolle

r 1



OIM

LED

1

OIM

LED

0

2+1 Single Topology

49

Plane 0

Plane 1

Plane 4

Plane 2

Plane 5

Plane 3

Plane 6

Plane 7


Fabric Placement with Three CRS-16 LCC

MultiModule Topology Example

SFC 0

SFC 1

SFC 2

SFC 3

SFC 4

SFC 5

SFC 6

SFC 7

SFC 8

SFC 9

SFC 1

0

SFC 1

1

SFC 1

2

SFC 1

3

SFC 1

4

SFC 1

5

SFC 1

6

SFC 1

7

SFC 1

8

SFC 1

9

SFC 2

0

SFC 2

1

SFC 2

2

SFC 2

3

She

lf Co

ntro

ller 0

Sh

elf C

on

trolle

r 1



OIM

LED

1

OIM

LED

0

Dual FCC Configuration

SFC 0

SFC 1

SFC 2

SFC 3

SFC 4

SFC 5

SFC 6

SFC 7

SFC 8

SFC 9

SFC 1

0

SFC 1

1

SFC 1

2

SFC 1

3

SFC 1

4

SFC 1

5

SFC 1

6

SFC 1

7

SFC 1

8

SFC 1

9

SFC 2

0

SFC 2

1

SFC 2

2

SFC 2

3

She

lf Co

ntro

ller 0

Sh

elf C

on

trolle

r 1



OIM

LED

1

OIM

LED

0

SFC 0

SFC 1

SFC 2

SFC 3

SFC 4

SFC 5

SFC 6

SFC 7

SFC 8

SFC 9

SFC 1

0

SFC 1

1

SFC 1

2

SFC 1

3

SFC 1

4

SFC 1

5

SFC 1

6

SFC 1

7

SFC 1

8

SFC 1

9

SFC 2

0

SFC 2

1

SFC 2

2

SFC 2

3

She

lf Co

ntro

ller 0

Sh

elf C

on

trolle

r 1



OIM

LED

1

OIM

LED

0

Single FCC Configuration

Legend :

50

Plane 0

Plane 1

Plane 2

Plane 3 Plane 7

Plane 6

Plane 5

Plane 4


CRS-3 LC/Fabric(MultiChassis) connectivity

S2 Links S1 Links

S2 Fabric Card S13 Fabric Card S13 Fabric Card

LC

LC1

Seal Links IngressQ

S3

FabricQ

FabricQ

FabricQ

FabricQ

LC1

S1

S1

S3 S3 Links

S2

S2

S2

51


Failure detection and System Recovery

The CRS system works towards detecting different type of failures in the system and take corrective action. Some of these failures could be:

Fabric Failure

Board Failure

Cable Failure

ASIC failure for Fabric, RP or Line card.

ASIC errors like ECC, Pairity or other miscellaneous errors.

Any single node non fabric failure e.g. RP, DRP, MSC.

Link errors and recovery

Rack failure for LCC

Rack failure for FCC

52


Failure detection and System Recovery Contd...

Fabric failure

‒ The fabric board failures or cable failures are detected and reported by the fabric software and corrective action will be taken based on the severity of the failure in terms of

Plane MCAST_DOWN

All unicast traffic should pass through the plane except for destinations in the rack where the failure is based. This gives an oppurtunity for the system admins to recover from the malfunctioning board and cable and then bring the plane back into full action.

Plane DOWN

If the failure of fabric board or cables impacts the whole plane (e.g. An S2 board) , the plane is brought down completely so that traffic can be redirected through other planes till this failure is corrected and plane is functional again.

Note: Extensive FMEA testing has been done on all CRS-3 boards to detect, report and take corrective action of any possible board failure.

ASIC Failures and errors recovery

53


CRS MultiChassis Switch Fabric Details

MC Optical interconnects and cabling

MC SFC types (S13 and S2)

MC Fabric Topologies (Single module vs. Multi Module)

54


CRS Switch Fabric Interconnect Optical Interconnect

FIBER BUNDLE

‒12 fibers per ribbon cable

‒6 ribbon cables per bundle = 72 fibers per Array cable

Multiple cables Between LCC Chassis and FCC Chassis (100m max)

Array cable connections on an S13 card

55


CRS Array Cables Details

Array Cables connect FCC to the LCC

24 Array cables required for each LCC (3 per fabric plane x 8 fabric planes)

Array Cable composed of individual Fibers

Each cable consists of 6 ribbon cables with 12 fibers each

10m, 15m, 20m, 25m, 30m, 40m, 50m,60m,70m, 80m, 90m,100m length variants

Terminated at each end with a Square keyed Connector

56


Array Cables

Turn Radius collar attachment

57


CRS S13 Switch Fabric Card (SFC) For CRS-1 S13 contains the 2 S1 and 4 S3 ASICs from the S123 Card now

with parallel optical devices (PODs)

For CRS-3 S13 contains 2 ASIC‘s (S1,S3 mode)

For CRS-1 top S1 and S3 elements service linecard shelf slots 0->7 Bottom S1

and S3 elements service slots 8->15 plus 2 RP slots

For CRS-3 all LC and RP are connected to both SEA S3 ASIC‘s, three

connections to each Asic from each LC.

Each POD terminates a Fiber ribbon consisting of 12 unidirectional links each

operating at 2.5Gbps for CRS-1 and 5Gbps for CRS-3

The POD perform electrical to optical or optical to electrical conversions for

data for transmission over multimode fiber (850 nm), for distances of up to 100

meters. 6 PODs are used for the 72 fibers from S1 ASICs/12 PODs are

used for the 144 fibers from S3 ASICs

58


CRS S13 SFC Card

S13 Service Processor

S1

S1

3* Tx POD 3 Fiber Ribbons

3 Fiber Ribbons

S3

S3 6* Rx POD

6* Rx POD

6 Fiber Ribbons

3* Tx POD

6 Fiber Ribbons S3

S3

59


CRS S13 - Fiber to bulkhead mapping

S1

S1

S3

S3

S3

S3

S3 Fibers

S1 Fibers

60


Fiber mapping

61


CRS S2 Switch Fabric Card (SFC)


S2

S2

6* Rx POD

12* Tx POD

6 Fiber Ribbons

12 Fiber Ribbons

S2

S2

6* Rx POD

12* Tx POD

6 Fiber Ribbons

12 Fiber Ribbons

S2

S2

6* Rx POD

12* Tx POD

6 Fiber Ribbons

12 Fiber Ribbons

• S2 Card consists of PSU, Service Processor and 3*S2 sub-boards

• 6 S2 ASICs and 54 parallel optical devices (PODs) per card for CRS-1

• 3 S2 ASICs and 54 parallel optical devices (PODs) per card for CRS-3

• Each POD terminates 1 Fiber ribbon (containing 12 fibers)

• 6 PODs per sub-board are used for the 72 fibers from the S1 ASICs

• 12 PODs per sub-board are used for the 144 fibers to the S3 ASICs

Bo

ard 0

B

oard

1

Bo

ard 2

62


Fabric Chassis S2 Optical Interface Module (OIM)

• Passive device providing fiber X-connect function

• OIM distributes the fibers within each bundle to the S2 ASICs

• 9 Array cables per ‘single-wide’ OIM provides connectivity for up to 3 LCC chassis’

Larger size chassis’ deployments require cabling layout to be switched from vertical to horizontal

Note: OIM must be installed before S2 card insertion

63


CRS S13 S2 card interconnect for MC single

module (vertical) cabling


S2

S2

6* Rx POD

12* Tx POD

6 Fiber Ribbons

12 Fiber Ribbons

S2

S2

6* Rx POD

12* Tx POD

6 Fiber Ribbons

12 Fiber Ribbons

S2

S2

6* Rx POD

12* Tx POD

6 Fiber Ribbons

12 Fiber Ribbons

Rack 0 S13 Card Rack 1 S13 Card Rack 2 S13 Card

64


CRS MC Optical Interface Module

To LC rack # 0

• A ‘single-wide’ Optical Interface Module is mated to one S2 Switch Fabric Card

• This configuration can support up to 3 chassis’ for single module (vertical) cabling

To LC rack # 2

65


0 1 2 3 S13 Fabric Slots

0 1 2 3 4 5 6 7 8 LCC0 - Plane 0

11 10 9 8 7 6 5 4 3 2 1 0 S2 Fabric Slots

Single Module Mode

FCC Chassis

A0

A1

A2

LCC0

LCC1

LCC2

CRS MC Fabric Topology

LCC0 Chassis

rear view right-to-left rear view left-to-right

FCC - Plane 0

66


0 1 2 3 fabric Slots

LCC0 - Plane 0

LCC0 Chassis

A0

A1

A2

0 1 2 3 4 5 6 7 8

11 10 9 8 7 6 5 4 3 2 1 0 Slots

FCC Chassis

Multi Module Mode

LCC1 LCC0

LCC3 LCC2

LCC5 LCC4

LCC8

LCC6 LCC7


rear view left-to-right

FCC - Plane 0 rear view right-to-left

67


CRS Optical Interface Module – Fiber distribution

• SINGLE-WIDE OIM

• Individual fiber connections are ‘wired’ to all S2 ASICs

• 3 TX fiber ribbons (36 links in total) are used to connect from each of the S1 ASICs in the LCC to the S2 ASICs in the FCC

• 36 links go to the CRS-1 S2 board, with 6 links to each of the 6 S2 ASIC on the board

• 36 links go to the CRS-3 S2 board, with 12 links to each of the 3 S2 ASICs on the board

Rack0 Plane X S1->S2

68


CRS Optical Interface Module – Fiber distribution

• SINGLE-WIDE OIM

• Each S2 ASIC in CRS-1 FCC has 72 TX fiber links

• Each S2 ASIC in CRS-3 FCC has 144 TX fiber links

• 6 links are used to connect to each of the S3 ASICs

• 144 S2->S3 links per line card shelf per plane

• This configuration can service 12 S3 ASICs per plane

Rack0 Plane X S2->S3

69


CRS OIM (horizontal cabling)

To LC rack # 0

• For up to 9 chassis’, a single fabric plane spans 3 S2 SFC’s

To LC rack # 2

To LC rack # 8

To LC rack # 6

cabling arrangement as follows:

70


CRS MC Fabric Chassis - Cabling CONNECTION MAP

‒Based on configuration, a p2p connection map will be generated by the router

LED INSTALLATION AID

‒Flag misconnected fiber bundles

‒If only 1 fiber bundle is misconnected, identify correct plug hole (with some amount of

persistence)

‒5 states, 1 tri-color LED :

• OK (green)

• no signal (none)

• misconnected one cable (blinking red)

• misconnected more than one cable (red)

• signal fault (yellow)

• connect here (slow blinking green)

coresponds to blinking red

71


CRS MC 2+1 System (vertically) cabled

FCC LCC 0 LCC 1

8 S2 boards & OIMs

8 S13 boards

72


CRS MultiChassis Fabric – Data Path

IngressQ S1

S2

FabricQ S3

Linecard LC Midplane

S13 Card Bundle Fiber Module

S2 Card

1. Data segmented. Cells distributed over 8 planes

2. Load balance to the available S2s in plane

3. Switch cell to correct S3. Multicast is replicated here.

4. Switch cell to FabricQ. 5. Data reassembled into packets

LCC FCC fiber

73


• Each CRS-3 LC has 6* links to the S1 asic. So total 48 links (8 planes). Each TX Link has a capability of carrying 5G of raw traffic.

• Each CRS-1 LC has 4 connections to the S1 ASICs on the fabric plane (8 planes = 32 links). Each TX Link is capable of carrying 2.5G of raw traffic.

• Each CRS-1 RP has 2 connections to the Lower S1 ASICs on the fabric plane only (8 planes = 16 links). This is 2.5G as well.

Fabric Plane

Upper shelf

Lower shelf

Ingress LC

Ingress LC

RP

S1

S1 S3

S3

S2

S2

S2

Ingress CRS 16-slot LCC

74


CRS MC Fabric – Terminology

Link: ‒Data link between two fabric stages. Each link has two link ports, a transmitter and a receiver.

Bundle: ‒Cable between the LC and Fabric shelves. Each end terminates at a bundle port. Also known as Array cable.

Link port types Ingressqtx, s1rx ,s1tx, s2rx, s2tx, s3rx, s3tx and fabricqrx.

Link port inter-connect Ingressqtxs1rx, s1txs2rx, s2txs3rx, s3txfabricqrx.

75


Select from LCC0 – Plane 0 – A0

Remove Dust Caps

Connect one end into Bundle A0


76


Connect LCC0 – Plane 0 – Bundles A0, A1, A2

A1

A0

A2


77


A1

A0

A2

Connect bundle other end on FCC0 – Plane0


78


CRS MultiChassis Control Ethernet

MC control Ethernet functions

MC control Ethernet HW – Integrated Shelf Controller

MC control Ethernet connectivity

79


MS

C

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

Fan\F

AB

RIC

(rear)

Power Supplies

Air Intake

Air Exhaust (r)

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

RP

/FA

BR

IC(r

ear)

PL

IM

PL

IM

LC

PL

IM

MS

C

PL

IM

MS

C

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

Fan

\FA

BR

IC(r

ear)

Power Supplies

Air Intake

Air Exhaust (r)

PL

IM

PL

IM

PL

IM

PL

IM

PL

IM

RP

/FA

BR

IC(r

ear)

PL

IM

PL

IM

LC

PL

IM

MS

C

PL

IM

24 cables to each LC chassis

RP1

Air Exhaust (r)

Air Exhaust (r)

SC

-GE

-22

S

C-G

E-2

2

12 x S2

Power Supplies

12 x S2

RP0

Stacking link

LCC0 LCC1 FCC

Connected to OIM’s (rear) on FCC

Connected to S13 cards (rear) on LCC

2LCC +1FCC MC Component connectivity via SC

80


CRS-1 MultiChassis Control Ethernet

All communication from the Line card RPs to integrated switch is over the Control Ethernet

‒The integrated switch is not connected to the fabric

The Control Ethernet is used for many purposes

‒System Boot

‒Node availability (Heart beat) checks

‒All communication from the LCC to the FCC.

The Control Ethernet is redundant and must be connected in a fully meshed configuration to all active and standby RPs and SCs

‒2+1 Systems requires 9 cables – 8 RP to SCGE and 1 SCGE to SCGE

‒2+2 System requires 15 cables – 8 RP to SCGE and 6 SCGE to SCGE

‒2+4 System requires 36 cables – 8 RP to SCGE and 28 SCGE to SCGE

The Control Ethernet uses Spanning Tree (STP) to determine which paths to use for communication

81


Integrated Shelf Controller

Product Description:

–Fabric chassis houses 2 Shelf Controllers (SC) acting as Primary and Secondary

–Provides local management of fabric chassis components

Boot and initialization of the SFCs, the optical interface module LED (OIM-LED) card, alarms, power supplies, and fans

–Integrated GE Ethernet Switch Interface

Provides out of band inter-chassis control network

Full Mesh connectivity required for control Ethernet

–Two SC-22GE are provided for redundancy

Product ID:

CRS-FCC-SC-22GE

82


2+1 MC External GE Connections

SC-GE-22

SC-GE-22

SC0

SC1

RP0

RP1

RP0

RP1

SC0-RP GE Links

SC1-RP GE Links

SC-SC GE Links

Note there is still an FE link over the backplane on the FC between the 2 SC-

GE-22 cards

LCC0

LCC1

FCC0

83


2+2 MC External GE Connections

SC-GE-22

SC-GE-22

SC0

SC1

RP0

RP1

RP0

RP1

LCC0

LCC1

FCC0

SC-GE-22

SC-GE-22

SC0

SC1

FCC1 84


CRS MultiChassis Configuration

MC dSC

MC SW distribution

MC rack configuration

MC fabric plane topology configuration

85


CRS MC - Introducing the dSC

dSC = designated System Controller

dSC is responsible for overall system control, configuration and operation

Image download & synchronization to all devices in system is controlled by dSC

By default, dSC is the Primary RP in the first rack (LCC) that boots. If RP fails, Secondary RP assumes the role

If the LCC housing the dSC where to fail, today, MC system will reboot and dSC will come active on one of other LCC‘s connected to the system

Eventually, dSC functionality will be able to move between racks in a graceful manner

No specific configuration is required to become dSC

86


IOS-XR SW version on MC

dSC determines the IOS XR version on all components of the system

– Adding a new LCC with different IOS XR version will install the version running on the dSC

– Upgrading from single to MC – the FCC will install the IOS XR version running on the dSC

– Inserting new MSCs – same as on single chassis

87


CRS MC – configuration Rack numbers

RP/0/RP0/CPU0:CRS3#admin show run | i dsc

Building configuration...

dsc serial TBA10080000 rack 1


dsc serial TBA10120000 rack f0

• Assign a rack number to the S/N of each chassis • Serial numbers can be obtained from ‘sh diag chassis’ - From rommon with “dumpplaneeeprom” - From rear of the FCC and front of the LCC • Rack 0 or 1 = dSC (also an LCC) • Rack 1->127 = LCC • Rackf0-> f4 = FCC • Note: Rack numbers have to be unique Example:

88


CRS LCC / FCC Serial Number Location

Watchout ! It is on the rear Side of the Fabric Chassis, unlike line card chassis

Front side of the Linecard Chassis

LCC FCC

89


CRS MC Fabric – Config

Plane Topology configuration

Planes are not tied to specific slots in fabric rack

Admin-level config defines the slots each plane uses (and how big the plane is) controllers fabric plane 0

oim count 1

oim width 1

oim instance 0 location F0/SM0/FM

Count 1- All cables in plane connect to the same OIM. Count 3 - The cables from each LCC for that plane connect to different OIMs

Position of 1st card within the plane in fabric rack: plane 0 uses rack f0 slot sm0

90


CRS MC Fabric Configuration Example RP/0/RP0/CPU0:CRS3#admin show run

Building configuration...



dsc serial TBA10120000 rack F0

controllers fabric plane 0

oim count 1

oim width 1


!


oim count 1

oim width 1


!


oim count 1

oim width 1


!

[SNIP]


oim count 1

oim width 1


!

9 6 3 0

21 18 15 12

91


Troubleshooting CRS MC

MC Control Ethernet connectivity verification & statistics

MC Control Ethernet UDLD and spanning tree functions

MC Shelf Controller (SC) LED‟s

Monitoring MC fabric plane, bundles and links

92


CRS MC RP Connectivity

To verify the control ethernet connectivity on the RPs use: RP/0/RP0/CPU0:router(admin)#show controllers switch 0 ports location 0/RP0/CPU0

Ports Active on Switch 0

FE Port 0 : Up, STP State : FORWARDING (Connects to - 0/RP0)

FE Port 1 : Up, STP State : BLOCKING (Connects to - 0/RP1)

FE Port 2 : Up, STP State : FORWARDING (Connects to - 0/FC0)

FE Port 3 : Up, STP State : FORWARDING (Connects to - 0/FC1)

FE Port 4 : Up, STP State : FORWARDING (Connects to - 0/AM0)

FE Port 5 : Up, STP State : FORWARDING (Connects to - 0/AM1)

FE Port 6 : Down (Connects to - )

FE Port 7 : Down (Connects to - )

FE Port 8 : Up, STP State : FORWARDING (Connects to - 0/SM0)








GE Port 0 : Up, STP State : FORWARDING (Connects to - GE_0)

GE Port 1 : Up, STP State : FORWARDING (Connects to - Switch 1)

93


CRS MC SC Intra-rack Connectivity To verify the control ethernet connectivity on intra-rack switches on

the SC-GE-22s use: RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch 0 ports location F0/SC0/CPU0

FE Port 0 : Up, STP State : FORWARDING (Connects to - F0/SC0)

FE Port 1 : Up, STP State : BLOCKING (Connects to - F0/SC1)

FE Port 2 : Down (Connects to - F0/FC0)

FE Port 3 : Down (Connects to - F0/FC1)

FE Port 4 : Down (Connects to - F0/AM0)

FE Port 5 : Up, STP State : FORWARDING (Connects to - F0/AM1)

FE Port 6 : Up, STP State : FORWARDING (Connects to - F0/LM0)

FE Port 7 : Up, STP State : FORWARDING (Connects to - F0/LM1)

FE Port 8 : Down (Connects to - F0/SM0)

FE Port 9 : Up, STP State : FORWARDING (Connects to - F0/SM1)







GE Port 0 : Up, STP State : FORWARDING (Connects to - GE_0)

GE Port 1 : Up, STP State : FORWARDING (Connects to - Switch 1)

94


CRS MC RP UDLD

UDLD runs on links between RPs and SCs. Use the UDLD

CLI to find out who is connected – this is a nice extra

feature of UDLD

RP/0/RP0/CPU0:ios(admin)#show controllers switch udld location 0/rp0/CPU0 … Interface GE_Port_0

…

…

Current bidirectional state: Bidirectional

Current operational state: Advertisement - Single neighbor detected

…

Entry 1

---

Device name: nodeF0_SC0_CPU0

Port ID: Gig port# 13

Neighbor echo 1 device: 0_RP0_CPU0_Switch

Neighbor echo 1 port: GE_Port_0

…

95


CRS MC SC Intra-rack Connectivity

Use the UDLD CLI the same way as for RPs

RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch udld location F0/SC0/CPU0

Interface GE_Port_0

---

Port enable administrative configuration setting: Enabled

Port enable operational state: Enabled



Message interval: 7

Time out interval: 5

Entry 1

---

Expiration time: 15

Device ID: 1

Current neighbor state: Bidirectional

Device name: nodeF0_SC0_CPU0

Port ID: Gig port# 22

Neighbor echo 1 device: F0_SC0_CPU0_Switch

Neighbor echo 1 port: GE_Port_0

96


CRS MC SC Inter-rack Connectivity To verify the control ethernet connectivity on inter-rack

switches on the SC-GE-22s use: RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch inter-rack ports all location F0/SC0/CPU0

GE_Port_0 : Up

GE_Port_1 : Down

GE_Port_2 : Up

GE_Port_3 : Up

[SNIP]

GE_Port_17 : Down

GE_Port_18 : Down

GE_Port_19 : Down

GE_Port_20 : Down

GE_Port_21 : Up

97


CRS MC RP STP STP is run on links between RPs and SCs. Use the STP

CLI to find out spanning tree info

RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch stp location 0/RP0/CPU0

##### MST 0 vlans mapped: 2-4094

Bridge address 0011.9362.a26c priority 36864 (36864 sysid 0)

Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)

port GE_Port_0 path cost 0

Regional Root address 5246.48f0.20ff priority 32768 (32768 sysid 0)

internal cost 20000 rem hops 3

Operational hello time 1, forward delay 6, max age 8, txholdcount 6

Configured hello time 1, forward delay 6, max age 8, max hops 4

Interface Sts Role Cost Prio.Nbr Type

---------------- ---- ---- --------- -------- ------------------------------

##### MST 1 vlans mapped: 1

Bridge address 0011.9362.a26c priority 36865 (36864 sysid 1)


port GE_Port_0 cost 20000 rem hops 3


---------------- ---- ---- --------- -------- ------------------------------

FE_Port_1 FWD Desg 200000 128. 2 P2p

GE_Port_0 FWD Root 20000 128. 49 P2p

98


CRS MC SC Intra-rack Connectivity

Use the STP CLI the same way as for RPs

RP/0/RP0/CPU0:CRS-D(admin)#sh controllers switch stp location F0/SC0/CPU0


Bridge address 0800.453e.47b4 priority 36864 (36864 sysid 0)








--------------- ---- ---- --------- -------- ------------------------------


Bridge address 0800.453e.47b4 priority 36865 (36864 sysid 1)




---------------- ---- ---- --------- -------- ------------------------------

FE_Port_1 BLK Altn 200000 128. 2 P2p

GE_Port_0 FWD Root 20000 128. 49 P2p

99


CRS MC SC Inter-rack Connectivity Use UDLD CLI to find out who is connected

RP/0/RP0/CPU0:ios(admin)#show controllers switch inter-rack udld all location f0/sc0/CPU0

Interface Gig port# 0

---

Port enable administrative configuration setting: Enabled

Port enable operational state: Enabled



Message interval: 7


Entry 1

---

Expiration time: 14

Device ID: 1

Current neighbor state: Bidirectional

Device name: 0_RP0_CPU0_Switch

Port ID: GE_Port_0

Neighbor echo 1 device: nodeF0_SC0_CPU0

Neighbor echo 1 port: Gig port# 0

Message interval: 7


CDP Device name: BCM_SWITCH

Interface Gig port# 2

--- 100


CRS MC SC Inter-rack Connectivity Use STP CLI to find out STP information

RP/0/RP0/CPU0:ios(admin)#show controllers switch inter-rack stp location f0/sc0/cpu0


Bridge address 5246.48f0.20ff priority 32768 (32768 sysid 0)

Root this switch for the CIST



Interface Role Sts Cost Prio.Nbr Type

---------------- ---- --- --------- -------- ---------------------------


Bridge address 5246.48f0.20ff priority 32769 (32768 sysid 1)

Root this switch for MST1


---------------- ---- --- --------- -------- ---------------------------

GE_13 Desg FWD 20000 128. 14 P2p

GE_14 Desg FWD 20000 128. 15 P2p

GE_15 Desg FWD 20000 128. 16 P2p

GE_17 Desg FWD 20000 128. 18 P2p

GE_22 Desg FWD 20000 128. 23 P2p

101


CRS MC Show tech control-ethernet

Use this CLI to collect control ethernet logs for TAC

RP/0/RP0/CPU0:IOX(admin)#show tech-support control-ethernet file ..

…

…

…

102


CRS MC SC-GE-22 Counters Checking port statistics counters

RP/0/RP0/CPU0:ios(admin)#show controllers switch inter-rack statistics all brief

location f0/sc0/cpu0

Port Tx Frames Tx Errors Rx Frames Rx Errors

-----------------------------------------------------------

GE_Port_0 : 374423 0 1776848 0

GE_Port_1 : 251232 0 170742 0

GE_Port_2 : 857923 0 414409 0

GE_Port_3 : 239437 0 152772 0

GE_Port_4 : 166166 0 82031 0

GE_Port_5 : 0 0 0 0

<SNIP>

GE_Port_16 : 0 0 0 0

GE_Port_17 : 0 0 0 0

GE_Port_18 : 0 0 0 0

GE_Port_19 : 0 0 0 0

GE_Port_20 : 0 0 0 0

GE_Port_21 : 0 0 0 0

Intra-rack : 522072 0 293720 0

Stacking : 1482 0 0 0

Stacking : 0 0 0 0

103


CRS MC SC-GE-22 Counters

Clear statistics using

Per port:

RP/0/RP0/CPU0:ios(admin)#clear controller switch inter-rack statistics ports 0

location f0/sc0/cpu0

Or all ports:

RP/0/RP0/CPU0:ios(admin)#clear controller switch inter-rack statistics all location

f0/sc0/cpu0

104


CRS MC SC-GE-22 LEDs

The SC-GE-22 has LEDs on the front panel for every port. The LEDs can be

used to get information about the link such as

‒Green Link Up

‒Blinking Green Activity

‒Amber Port Error Disabled (by UDLD)

‒Off Link Down

‒What color is stp blocking state? – it still stays green. In blocking state you are still

receiving udld/stp packets. The amber condition only indicates a fault on the port

‒Note: Admin Shutdown of ports is not supported

105


CRS MC SC UDLD error messages

Whenever a port is disabled in the control network due to it being detected

as Uni-directional, a Syslog message is generated

The state of the port is displayed in ―show controller switch .. udld ..loc <>‖

CLI

The fiber and SFP should be checked on the port

You can try to bring-up the port and clear the error condition using

‒―clear controller switch [inter-rack] errdisable port …‖

Note that the port will be err-disabled again if the error condition still exists

106


CRS MC SC Spanning tree

The following is the default priority order of nodes to become the root of the network

‒F0/SC0

‒F0/SC1

‒F1/SC0

‒F1/SC1

‒F2/SC0

‒F2/SC1

‒...

‒we use bridgeIDs to compare – priorities are same and mac address determine. Higher number is lower priority for STP, lower number wins. Note we don‘t us a guma (globally unique mac add) so we generate it geographically and can set the above up. Note that ma comes from the location of the card, its not burnt into the board itself, its generated at runtime during bootup

The LCC RP‘s are given lower priority than all FCC SC‘s so they will not become the root of the network as long as a single SC is booted up

107


CRS MC Spanning tree – to see the root Use the following command to find the STP status on a

node. Verify that the root is the node you expect it to be RP/0/RP0/CPU0:ios(admin)#show controllers switch stp location f0/sc1/CPU0


Bridge address 0800.453e.468f priority 36864 (36864 sysid 0)








---------------- ---- --- --------- -------- ---------------------


Bridge address 0800.453e.468f priority 36865 (36864 sysid 1)




---------------- ---- --- --------- -------- ---------------------

FE_Port_0 Altn BLK 200000 128. 1 P2p

GE_Port_0 Root FWD 20000 128. 49 P2p

108


CRS MC Spanning tree

in a normal topology Verify that F0/SC0 inter-rack is the root of the network and all spanning

tree port states seem normal

Verify that other SCs in the system see F0/SC0 as the root of the network

and are designated on links connecting to RPs

Verify that the RP has selected one of the GE ports as its‘ root port and is

‗alternate‘ on the other GE port

Verify that one of the RPs is blocked on the FE link connecting the RPs

109


Verifying MC control network with “ping”

―Ping control-eth‖ from admin mode can be used for troubleshooting control-eth issues

RP/0/RP0/CPU0:CRS-D(admin)#ping control-eth location f0/sc0/cpu0

Src node: 513 : 0/RP0/CPU0

Dest node: 983553 : F0/SC0/CPU0

Local node: 513 : 0/RP0/CPU0

Packet cnt: 1 Packet size: 128 Payload ptn type: default (0)

Hold-off (ms): 1 Time-out(s): 2 Max retries: 5

DelayTimeout: 1Destination node has MAC addr 5246.480f.0201

Running CE node ping.

Please wait...

Src: 513, Dest: 983553, Sent: 1, Rec'd: 1, Mismatched: 0

Min/Avg/Max RTT (usecs): 1000/1000/1000

CE node ping succeeded for node: 983553

110


Monitoring CRS MC Fabric Operation Error counts

RP/0/RP0/CPU0:CRS(admin)#show controllers fabric plane all statistics

In Out CE UCE PE

Plane Cells Cells Cells Cells Cells

---------------------------------------------------------------------------

0 45088937799 45088965120 0 0 0

1 45088939239 45088966386 0 0 0

2 45088554913 45088582368 0 0 0

3 45088556270 45088583448 0 0 0

4 45088151158 45088178280 0 0 0

5 45088152592 45088179766 0 0 0

6 45089166942 45089194313 0 0 0

7 45089168302 45089194928 0 0 0

CE Cells – Correctable Error UCE – Uncorrectable Error PE – Parity Error

111


Monitoring CRS MC Fabric Operation High level fabric status

RP/0/RP0/CPU0:CRS(admin)#show controllers fabric plane all detail

Plane Admin Oper Down Total Down Id State State Flags Bundles Bundles ------------------------------------------------------ 0 UP UP 0 0 1 UP UP 0 0 2 UP UP 0 0 3 UP UP 0 0 4 UP UP 0 0 5 UP UP 0 0 6 UP UP 0 0 7 UP UP 0 0

Examples of Down flags P – plane admin down p – plane oper down C – card admin down c – card oper down

112


Check the CRS MC fabric bundles All fibers are UP on each bundle ?

RP/0/RP0/CPU0:ios(admin)#sh controllers fabric bundle all detail

Flags: P - plane admin down, p - plane oper down

C - card admin down, c - card oper down

L - link port admin down, l - linkport oper down

A - asic admin down, a - asic oper down

B - bundle port admin Down, b - bundle port oper down

I - bundle admin down, i - bundle oper down

N - node admin down, n - node down

o - other end of link down d - data down

f - failed component downstream

m - plane multicast down

Bundle Oper Down Plane Total Down Bundle Bundle

R/S/M/P State Flags Id Links Links Port1 Port2

----------------------------------------------------------------------

F0/SM0/FM/0 UP 3 72 0 F0/SM0/FM/0 0/SM3/SP/0






F0/SM0/FM/6 DOWN bo 3 72 72 F0/SM0/FM/6 2/SM3/SP/0



[SNIP]

113


Check the CRS MC fabric DOWN links F0/SM6/FM/1 UP 1 72 1 F0/SM6/FM/1 0/SM1/SP/1

(admin)#sh controllers fabric link port s2tx all | i UP.*DOWN.*SM1/SP/1

F0/SM6/SP/1/61 UP DOWN do 1/SM1/SP/1/25 F0/SM6/3 1/SM1/0

(admin)#sh controllers fabric link port s3rx all | i 1/SM1/SP/1/25

1/SM1/SP/1/25 UP DOWN l F0/SM6/SP/1/61 1/SM1/0 F0/SM6/3

(admin)#sh controllers fabric link port s2tx F0/SM6/SP/1/61 detail

[SNIP]

Sfe Port Admin Oper Down Sfe BP Port BP Other

R/S/M/A/P State State Flags Role Role End

----------------------------------------------------------------

F0/SM6/SP/1/61 UP DOWN do 1/SM1/SP/1/25

Connection Details for s2tx/F0_SM6_SP,0x1,0x3d

---------------------------------------

Type: Inter-chassis bundle

Near-end bundle port: bport/F0/SM6/3 ribbon 3 fiber 1

Far-end bundle port : bport/1/SM1/0 ribbon 2 fiber 1

HBMT pin name : P6L2_1

Fabric group offset : 0

Fabric group : 2

(admin)#sh controllers fabric link port s3rx 1/SM1/SP/1/25 detail

Start to clean here

114


Check the CRS MC Bundle Statistics RP/0/RP0/CPU0:ios(admin)#show controllers fabric bundle port all statistics

Total racks: 3

Rack 0:

Bundle Port In Out CE UCE PE

R/S/M/P Cells Cells Cells Cells Cells

--------------------------------------------------------------------------------

0/SM0/SP/0 10032966 2237486 1484566 6 0

0/SM0/SP/1 4571827 2237495 776173 0 0

Rack 1:



--------------------------------------------------------------------------------

1/SM0/SP/0 7198601 10115782 7015647 910 0

1/SM0/SP/1 6978240 7649778 92393711 13972 0

Rack F0:



--------------------------------------------------------------------------------

F0/SM4/FM/0 159802 236890 0 0 0

F0/SM4/FM/1 159766 236825 21 10 0

115


More useful commands - CRS MC fabric

Fabric

‒ (admin)#show controllers fabric rack all detail

‒ (admin)#show controllers fabric plane all detail

‒ (admin)#show controllers fabric connectivity all detail

‒ (admin)#show controllers fabric plane all statistics

116


Cisco CRS MultiChassis is managed as a system.

117


Recommended Reading

118


Evaluation Forms

Please fill out evaluation forms and surveys

BRKARC-3002

119


Complete Your Online

Session Evaluation Give us your feedback and you

could win fabulous prizes.

Winners announced daily.

Receive 20 Passport points for each

session evaluation you complete.

Complete your session evaluation

online now (open a browser through

our wireless network to access our

portal) or visit one of the Internet

stations throughout the Convention

Center.

Don‘t forget to activate your

Cisco Live Virtual account for access to

all session material, communities, and

on-demand and live activities throughout

the year. Activate your account at the

Cisco booth in the World of Solutions or visit

www.ciscolive.com.

120

http://www.ciscolive.com


Final Thoughts

Get hands-on experience with the Walk-in Labs located in World of

Solutions, booth 1042

Come see demos of many key solutions and products in the main Cisco

booth 2924

Visit www.ciscoLive365.com after the event for updated PDFs, on-

demand session videos, networking, and more!

Follow Cisco Live! using social media:

‒ Facebook: https://www.facebook.com/ciscoliveus

‒ Twitter: https://twitter.com/#!/CiscoLive

‒ LinkedIn Group: http://linkd.in/CiscoLI

121

http://www.ciscolive365.com/

https://www.facebook.com/ciscoliveus

https://twitter.com/

http://linkd.in/CiscoLI

© 2012 Cisco and/or its affiliates. All rights reserved. Presentation_ID Cisco Public


SC to MC conversion can be done : online (most customers) & offline

MC Downgrade to a SC is not supported

Online Conversion :

SC is live & carrying production traffic

CRS Fabric can route line rate IMIX traffic even with 7 planes UP

SC to MC online migration uses the above concept to migrate each plane one by one (S123 boards to S13 boards)

Though done in maintenance window, the control plane traffic is still live

Offline Conversion :

Customers who do not want to change metrics config to divert the traffic

Easier process if the operational folks in the POP & N/W admins in the Central Site

Least risky

Appendix – Typical Migration Steps SC to MC

Types of Migration : Online vs Offline

123


Appendix A – Case Study 1:

Offline SC to MC migration - Network Topology SBCR (Super Block Core Router)

BCR (Block Core Router)

AER (Area Edge Router)

SSE

L2SW

CO

RE

EDG

E A

CC

ESS

Aggregation routers of East connecting to West

Aggregation routers of each region

Aggregation routers of each prefecture

Multi-Service customer edge router

L2 aggregation switch

MC Installation

Site A

Site B

124


Appendix A – Case Study 1

MC Migration Plan Take one week maintenance window for one site

Setup 1+1 MC config for system verification

No live migration (Off-line upgrade from single chassis to multi chassis)

SSE SSE

Site A Site B

SSE SSE

Site A (Primary) Site B (Secondary)

125



MC Migration Steps

SSE SSE SSE SSE

Site A (Primary) Site B (Secondary) Site A Site B

SSE SSE

Site A Site B

SSE SSE

Site A Site B

1 2

3 4

Install new LCC and FCC Setup 1+1 MC configuration Verify 1+1 MC system

Offline

Single to Multi Migration

dSC FCC

dSC FCC ndSC

126



MC Migration Steps – contd.

SSE SSE

Site A Site B

SSE SSE

Site A Site B

SSE SSE

Site A Site B

5 6

7 8

Install new LCC and FCC Setup 1+1 MC configuration Verify 1+1 MC system

Offline

Single to Multi Migration

SSE SSE

Site A Site B

Live dSC FCC

dSC FCC ndSC

127



MC Migration Scenario – contd.

9 SSE SSE

Site A Site B

Live

128



MC Installation Procedure – summary

LCC0 FCC0 LCC1

dSC

LCC0 FCC0 LCC1

dSC

MC 1+1

MC 2+1

Exisiting LCC

First build up a MC 1+1 (LCC Rack num = 1) and test it offline. Step1. LCC1 Setup Step2. LCC1 Turboboot Step3. FCC0 Bootup Step4. Connecting Array cables Step 5. Then connect the existing LCC(Rack Num = 0) into

MC 1+1 offline (more details in next slide)

Assumption: LCC0 is the Line Card Chassis that is on and in operation. LCC1 is the new Line Card Chassis. FCC0 is the new Fabric Chassis. LCC0 will NOT be touched when building a Rack 1 1+1 Multi-Chassis.

129



MC Installation Procedure - detail

LCC0 FCC0 LCC1

MC 1+1 Exisiting LCC

Step5. Connecting 1+1 MC into SC offline -Configure SC with SN of LCC1 & FCC -Configure LCC1 to be in install-mode -Bring down 1+1 MC -Replace all S123 boards to S13 & connect LCC0 to FCC -Connect the CE ports of LCC0 & 1 on FCC -Turn ON entire MC & allow it to bake -Ensure the Fabric is baked & ready -Run basic tests & ensure HW is fine on LCC1 -Remove “install-mode” from LCC1 -MC migration complete

XR RUN 3.6.1

XR RUN 3.6.1

Migrate S123 to S13 & connect cables

Turn ON

Verify LCC1 operation

Configure SN of LCC1 & FCC, install-mode

Remove LCC1 from install-mode

XR RUN 3.6.1

Verify MC operation

Power OFF Power OFF

Connect CE port Connect CE port Connect CE port

Power OFF

Turn ON Turn ON

130


Appendix B – Case Study 2: Online SC to MC

migration: MC Installation Procedure – summary

LCC0 FCC0 LCC1

dSC

LCC0 FCC0 LCC1

dSC

MC 1+1

MC 2+1

Exisiting LCC

First build up a MC 1+1 (LCC Rack num = 1) and test it offline. Step1. LCC1 Setup Step2. LCC1 Turboboot Step3. FCC0 Bootup Step4. Array Cable Connecting Step 5. Then connect the existing LCC(Rack Num = 0) into

MC 1+1 online (more details in next slide)

Assumption: LCC0 is the Line Card Chassis that is on and in operation. LCC1 is the new Line Card Chassis & will be put in install-mode FCC0 is the new Fabric Chassis. LCC0 will NOT be touched when building a Rack 1 1+1 Multi-Chassis.

131


Appendix B – Case Study 2

MC Installation Procedure - detail

LCC0 FCC0 LCC1

MC 1+1 Exisiting LCC

Step5. Connecting 1+1 MC into SC online -Bring down 1+1 MC -Connect the CE ports of LCC0 & 1 on FCC -Configure SC with SN of LCC1 & FCC -Configure LCC1 to be in install-mode -Turn ON FCC & allow it to bake -Ensure the Fabric is baked & ready -Migrate S123 boards on SC to S13 plane by plane -Once the SC is converted into LCC0 with all S13’s -Turn ON the LCC1 & allow it to bake -Run basic tests & ensure HW is fine on LCC1 -Remove “install-mode” from LCC1 -MC migration complete

XR RUN 3.6.1

XR RUN 3.6.1

Migrate S123 to S13 plane by plane

Turn ON FCC --> Fabric ready

Verify LCC1 operation

Turn ON LCC1

Configure SN of LCC1 & FCC, install-mode

Remove LCC1 from install-mode

XR RUN 3.6.1

Verify MC operation

Power OFF Power OFF

Connect CE port Connect CE port Connect CE port

132


Appendix C – FCC Physical Install notes

Optical Interface Module (OIM) - Rear

Back View

*Important: Always insert

OIM before inserting

corresponding S2 Fabric

Board. When removing

OIM, disengage S2 Fabric

Board first.

OIM

133


Appendix C – Physical Install notes

Tips for handling S2 Fabric Board

S2 Fabric Board is the longest and heaviest board.

Take care when carrying the board and swinging in an arc, not to hit the optical connectors.

Recommended hand placement when carrying the board.

Insert after OIM module.

134



Watch out for Bend Radius in Cable Trays

135


Appendix C – Physical Install notes Optics

Troubleshooting/Cleaning

S13 Manual Optics Cleaning

optic adapter

136



Optic Troubleshooting/Cleaning

OIM Manual Optics Cleaning

137

Documents

Cisco CRS-3 Carrier Routing System MultiChassis Overviewd2zmdbbm9feqrf.cloudfront.net/2012/usa/pdf/BRKARC-3002.pdf · Cisco CRS-3 Carrier Routing System MultiChassis Overview Session