T Series Core Router Architecture Review (Whitepaper)

White Paper

Copyright © 2012, Juniper Networks, Inc. 1

T SerIeS Core rouTer ArChITeCTure overvIewhigh-end Architecture for Packet Forwarding and Switching

2 Copyright © 2012, Juniper Networks, Inc.

White Paper - T Series Core Router Architecture Overview

Table of Contentsexecutive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Nine Years of Industry-Leading Core routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Forwarding Plane Design objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Architectural Framework and Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

PIC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

FPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Switch Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

routing engine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

T Series Forwarding Path Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Data Flow Through the 3D FPC on the T4000 router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Data Flow Through the enhanced Scaling (eS) FPCs on T Series routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

T Series Switch Fabric Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Switch Fabric operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Multichassis Switch Fabric (TX Matrix and TX Matrix Plus) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

CLoS Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Juniper Networks Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Multichassis System Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

About Juniper Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

List of FiguresFigure 1. T Series scaling by slot and chassis (including the JCS1200) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3

Figure 2. Scalable performance in multiple dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Figure 3. T Series routing platform architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 4. T Series router components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Figure 5. Data flow through the T4000 router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Figure 6. PFe and switch fabric using the T Series chipset (“to fabric” direction) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7

Figure 7. Packet flow from fabric to egress PFe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Figure 5. Five switch fabric planes for the T Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Figure 6. Multicast and unicast each utilizing fabric queues in T Series routers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .10

Figure 7. Three-stage CLoS topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Figure 8. T Series multichassis switch fabric planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Figure 9. Multichassis system high-level overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Figure 10. Switch-card chassis to line-card chassis interconnections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13



Executive SummaryJuniper Networks® T Series Core routers have been in production since 2002, with the introduction of the Juniper

Networks T640 Core router. Since that time, T Series routers have evolved to maintain an unequivocal industry lead in

capacity (slot, chassis, and system) and operational efficiencies in power and usability. Maintaining this standard has

in part been possible due to design decisions made with the very first T Series system.

Nine Years of Industry-Leading Core RoutingIn April 2002, Juniper Networks began shipping the first T Series routing platform: the T640 Core router. The T640 is a

carrier-class, multichassis-capable core routing platform that supports high-density 10 Gbps (oC-192c/STM-64 and 10

Gbps Gigabit ethernet) to 40 Gbps (oC-768c/STM-256 and 40 Gbps Gigabit ethernet) interfaces.

In July 2002, Juniper Networks began shipping the second T Series platform: the Juniper Networks T320 Core router.

The T320 is a carrier-class, single-chassis core router that has a smaller form factor than the T640 and a lower entry

price point.

In 2004, Juniper offered the first multichassis core routing system with the Juniper Networks TX Matrix, supporting 3.2

Tbps in a four-chassis system.

Then in 2007, Juniper announced and released the industry’s first 100 Gbps/slot system in the Juniper Networks T1600

Core router, a multichassis-capable routing platform that was designed—taking advantage of the original switch plane

architecture of the T640 —to be upgradeable in service from the T640.

In 2009, Juniper introduced the Juniper Networks TX Matrix Plus, a central switching and routing element that connects

up to 4 T1600 routing chassis into a single routing entity: a 6.4 Tbps system.

In 2011, Juniper introduced the 350 Gbps/slot-capable system, Juniper Networks T4000 Core router, which has

significantly extended the T Series system capacity, and allowed customers’ investment in the T Series to be protected

by upgrading existing T640 and T1600 systems to the T4000.

All T Series platforms use Juniper Networks Junos® operating system and T Series ASICs to provide the ease of use,

performance, reliability, and feature richness that service providers have come to expect from all Juniper Networks

products. The T Series Core routers—T320, T640, T1600, T4000, TX Matrix, and TX Matrix Plus—provide the

ingredients for high-end and core networks of the future, especially when controlled by the Juniper Networks JCS1200

Control System.

Figure 1 illustrates the industry-leading scaling characteristics of the T Series on the forwarding and control planes.

Figure 1. T Series scaling by slot and chassis (including the JCS1200)

This paper provides a technical introduction to the architecture of T Series Core routers, both single-chassis and

multichassis systems. It describes the design objectives, system architecture, packet forwarding architecture, single-

chassis switch fabric architecture, and multichassis switch fabric architecture.

For an explanation of the core virtualization capabilities available with the integration of the JCS1200, see the

references section in Appendix A, particularly the paper entitled Virtualization in the Core of the Network at www.juniper.net/us/en/local/pdf/whitepapers/2000299-en.pdf.

Single Chassis

JCS1200

T640 Ports40 10 Gbps8 40 Gbps

40 Gbps/slot 2002

Single Chassis T1600 Ports80 10 Gbps16 40 Gbps8 100 Gbps

100 Gbps/slot 2007

MultichassisTX Matrix Ports160 10 Gbps32 40 Gbps

2004

Multichassis

TX Matrix Plus Ports 320 10 Gbps32 64 40 Gbps 32 100 Gbps

2009

Single Chassis T4000 Ports208 10 Gbps16 40 Gbps16 100 Gbps

350 Gbps/slot 2011



Forwarding Plane Design Objectives Juniper’s architectural philosophy is based on the premise that multiservice networks can be viewed as three-

dimensional systems containing forwarding, control, and service dimensions:

• Forwarding (“how you move the bits”)

• Control (“how you direct the bits”)

• Software/service (“how you monetize the bits”)

For a network to be a converged, scalable, packet core infrastructure, it must scale in all these dimensions (Figure 2):

Figure 2. Scalable performance in multiple dimensions

In the forwarding plane, traffic growth is the key driver of the core router market. As the global economy becomes

increasingly networked and dependent upon the communications infrastructure, traffic rates continue to balloon—

growing 70 to 80 percent a year by most estimates—and high-density core routing remains critical.

Furthermore, the importance of the control plane cannot be overlooked. A router’s control plane must scale to

accommodate ever-growing routing and forwarding tables, service tunnels, virtual networks, and other information

related to network configuration and management.

Finally, the importance of the service plane is brought to bear when considering the requirements of an increasingly

disparate and global marketplace. These changing market dynamics create pressure for greater network innovation

and a much deeper integration between applications and the network.1.

T Series routing platforms are designed with this philosophy in mind at all times—the main focus of this paper is on the

forwarding plane of the router. The T Series routing platforms were developed to support eight key design objectives:

• Packet forwarding performance

• Bandwidth density

• IP service delivery

• Multichassis capability

• high availability (hA)

• Single software image

• Security

• Power efficiency

Juniper Networks leads the industry in all of these categories, as the white papers in the references section illustrate.

Forwarding: T1600

Co

ntr

ol:

JC

S12

00

Service: P

SDP



Architectural Framework and ComponentsFigure 3 illustrates the high-level architecture of a T Series routing platform. A T Series platform uses a distributed

architecture with packet buffering at the ingress Packet Forwarding engine (PFe) before the switch fabric, as well as

packet buffering at the egress PFe before the output port.

As a packet enters a T Series platform from the network, the ingress PFe segments the packet into cells, the cells are

written to ingress memory, a route lookup is performed, and the cells representing the packet are read from ingress

memory and sent across the switch fabric to the egress PFe. when the cells arrive at the egress PFe, they are written

to the second memory; an egress lookup is performed; the cells representing the packet are read out of the second

memory, reassembled into a packet, and transmitted on the output interface to the network.

Figure 3. T Series routing platform architecture

Drilling more deeply into the components, a T Series system consists of four major components: PICs, Flexible PIC

Concentrators (FPCs), the switch fabric, and one or more routing engines (Figure 4).

Figure 4. T Series router components

These components are discussed in more detail in the following sections.

PICThe PICs connect a T Series platform to the network and perform both physical and link-layer packet processing. They

perform all of the functions that are required for the routing platform to receive packets from the network and transmit

packets to the network. Many PICs, such as the Juniper IQ PICs, also perform packet processing.

FPCeach FPC can contain one or more PFes. For instance, a T4000 240 Gbps slot supports two 120 Gbps PFes. Logically,

each PFe can be thought of as a highly integrated packet-processing engine using custom ASICs developed by Juniper

Networks. These ASICs enable the router to achieve data forwarding rates that match fiber optic capacity. Such high

forwarding rates are achieved by distributing packet-processing tasks across this set of highly integrated ASICs.

when a packet arrives from the network, the ingress PFe extracts the packet header, performs a routing table lookup

and any packet filtering operations, and determines the egress PFe connected to the egress PIC. The ingress PFe

forwards the packet across the switch fabric to the egress PFe. The egress PFe performs a second routing table

lookup to determine the output PIC, and it manages any egress class-of-service (CoS) and quality-of-service (QoS)

specifications. Finally, the packet is forwarded to the network.

Packet Processing/Bu�er


Switch





PIC

FPCSwitching

Planes

PIC

FPC

PIC

FPC

PIC

FPC

Routing EngineRouting Engine



Switch FabricThe switch fabric provides connectivity between the PFes. In a single-chassis system, the switch fabric provides

connectivity among all of the PFes residing in the same chassis. In a multichassis system, the switch fabric provides

connectivity among all of the PFes in the different chassis of the routing node cluster. In a single-chassis or a multichassis

system, each PFe is considered to be logically contiguous to every other PFe connected to the switch fabric.

Routing EngineThe routing engine executes the Junos oS and creates the routing tables that are downloaded into the lookup ASICs

of each PFe. An internal ethernet connects the routing engine to the other subsystems of a T Series platform. each

subsystem includes one or more embedded microprocessors for controlling and monitoring the custom ASICs, and

these microprocessors are also connected to the internal ethernet.

T Series Forwarding Path ArchitectureThe function of the PFe can be understood by following the flow of a packet through the router: first into a PIC, then

through the switching fabric, and finally out another PIC for transmission on a network link. Generally, the data flows

through the PFe as follows:

• Packets enter the router through incoming PIC interfaces, which contain controllers that perform media-specific

processing (and optionally intelligent packet processing).

• PICs pass the packets to the FPCs, where they are divided into cells and are distributed to the router’s buffer memory.

• The PFe performs route lookups, forwards the cells over the switch fabric to the destination PFe, reads the cells from

buffer memory, reassembles the cells into packets, and sends them to the destination port on the outgoing PIC.

• The PIC performs encapsulation and other media-specific processing, and it sends the packets out into the network.

The use of highly integrated ASICs is critical to the delivery of industry-leading forwarding performance and

packet processing. The T Series chipset is specifically designed and developed to comprise the state of the art in

carrier-class routers.

Data Flow Through the 3D FPC on the T4000 Router 3D refers to the newer Type 5 FPC introduced along with the T4000 system. To ensure the efficient movement of data

through the 3D FPC on the T4000 router, the router is designed so that ASICs on the hardware components handle the

forwarding of data. Data flows through the 3D FPC on the T4000 router in the following sequence (see Figure 5):

Figure 5. Data flow through the T4000 router

PacketsIn

SwitchFabric

Lookup andPacket-

Processing ASIC

SwitchInterface

ASIC

LAN/WANInterface,

Bu�ering ASIC

PIC

Midplane

PacketsOut

SwitchFabric

Lookup andPacket-

Processing ASIC

SwitchInterface

ASIC

LAN/WANInterface,

Bu�ering ASIC

PIC



1. Packets arrive at an incoming PIC interface.

2. The PIC passes the packets to the FPC, where the interface ASIC does pre-classification and sends the packet

header to the Packet-Processing ASIC that performs Layer 2 and Layer 3 lookup. The Packet-Processing ASIC is

also capable of doing Layer 4 through Layer 7 packet processing.

3. The Interface ASIC receives the modified header from the Packet-Processing ASIC and updates the packet and

divides it into 64-byte cells.

4. The Interface ASIC sends these 64-byte cells to the Switch Fabric via the Switch Interface ASIC-facing Switch

Fabric, unless the destination is on the same Packet Forwarding engine. In this case, the Interface ASIC sends

packets to the outgoing port without passing them through the Switch Fabric.

5. The Interface ASIC sends bandwidth requests through the Switch Fabric to the destination port.

6. The destination Interface ASIC sends bandwidth grants through the Switch Fabric to the originating Interface ASIC.

7. upon receipt of each bandwidth grant, the originating Interface ASIC sends a cell through the Switch Fabric to the

destination Packet Forwarding engine.

8. The destination Interface ASIC receives cells from the Switch Fabric, reorders the data received as cells and

reassembles into packets, and passes the header to the Packet-Processing ASIC.

9. The Packet -Processing ASIC performs the route lookup, adds Layer 2 encapsulation, and sends back the header to

the Interface ASIC.

10. The Interface ASIC appends the modified header received from the Packet-Processing ASIC and sends the packets

to the outgoing PIC interface.

11. The outgoing PIC sends the packets out into the network.

Data Flow Through the Enhanced Scaling (ES) FPCs on T Series Routers This chipset includes the following ASICS:

• L2/L3 Packet-Processing ASIC (L2/3)

• Switch Interface ASIC

• T-Series Lookup Processor ASIC

• Queuing and Memory Interface ASIC (Q&M)

The T Series ASICs leverage the conceptual framework, many of the building blocks, and Juniper Networks extensive

operational experience—gained from the Juniper Networks M Series Multiservice edge routers chipset.

Figure 6 demonstrates data flow through a T Series routing node by illustrating how the T Series chipset is arranged to

implement a single instance of a PFe.

Figure 6. PFE and switch fabric using the T Series chipset (“to fabric” direction)

Layer 1ASIC

l2/L3Packet

Processing

SwitchInterface

ASIC

SwitchInterface

ASIC

Layer 1ASIC

l2/L3Packet

Processing

T SeriesLookup

Processor

Queuingand Memory

Interface ASIC

WAN

FPCFPCPIC SIB

Fabric

WAN

5

6

4

3

21



In this example, data flows in the following sequence:

1. Packets enter through an incoming PIC, which contains the Layer 1 interface chips, and are passed to the PFe on the

originating FPC. The PIC is connected to the PFe on the FPC via a high speed link (hSL).

2. The Layer 2/Layer 3 packet-processing ASIC parses the packets and divides them into cells. In addition, a behavior

aggregate (BA) classifier determines the forwarding treatment for each packet.

3. The network-facing Switch Interface ASIC places the lookup key in a notification and passes it to the T Series

Lookup Processor.

4. The Switch Interface ASIC also passes the data cells to the Queuing and Memory Interface ASICs for buffering

on the FPC.

5. The T Series Lookup Processor performs the route lookup and forwards the notification to the Queuing and Memory

Interface ASIC. In addition—if configured—filtering, policing, sampling, and multifield classification are performed at

this time.

6. The Queuing and Memory Interface ASIC sends the notification to the switch-fabric-facing Switch Interface ASIC,

which issues read requests to the Queuing and Memory Interface ASIC to begin reading data cells out of memory.

The destination FPC is shown in the following figure.

Figure 7. Packet flow from fabric to egress PFE

7. Cells representing packets are received from the Switch Interface ASIC connected to the fabric. The cells are

handed off to the Queuing and Memory Interface ASIC for buffering. Lookup keys and data pointers are sent to the

T Series Lookup Processor.

8. The T Series Lookup Processor performs the route lookup and forwards the notification to the Queuing and Memory

Interface ASIC, which forwards it to the network-facing Switch Interface ASIC.

9. The Switch Interface ASIC sends requests to the Queuing and Memory Interface ASIC to read the data cells out of

memory, and it passes the cells to the Layer 2/Layer 3 packet-processing ASIC.

10. The Layer 2/Layer 3 packet-processing ASIC reassembles the cells into packets, performs the necessary Layer 2

encapsulation, and sends the packets to the outgoing PIC. Queuing policy and rewrites occur at this time on the

egress router.

11. The PIC passes the packets into the network.

The T Series chipset provides hardware-based forwarding performance and service delivery for IPv4 (unicast and

multicast), IPv6 (unicast and multicast), and MPLS, while also having the flexibility to support other protocols in

the future. The chipset was designed to provide the functionality needed for the development of single-chassis and

multichassis systems.

Layer 1ASIC

l2/L3Packet

Processing

SwitchInterface

ASIC

SwitchInterface

ASIC

Layer 1ASIC

l2/L3Packet

Processing

T SeriesLookup

Processor

Queuingand Memory

Interface ASIC

WAN

FPCFPCSIB

Fabric

WAN

8

97

10 11



T Series Switch Fabric ArchitectureThe N+1 redundancy of the T Series switch fabric architecture is what has led to the in-service upgrade capability and

the flexibility of scaling the T Series in the multichassis dimension since its inception. For example, just by swapping

the power entry modules and the switch interface boards (using the N+1 redundancy), service providers can upgrade a

T640 Core router to a T1600 Core router without disrupting service or changing customer-facing interfaces.

Furthermore, connecting a T1600 platform to a TX Matrix Plus system is also a smooth in-service process, involving

upgrading the switch fabric boards and the control boards on the T1600 and then connecting the T1600 to the central

switching element—the TX Matrix Plus.

The five planes of the T Series switch fabric (Figure 5) are implemented using four operationally independent, parallel

switch planes (Labeled A through D) that are simultaneously active and an identical fifth plane (Labeled e) that acts

as a hot spare to provide N+1 redundancy.

Figure 5. Five switch fabric planes for the T Series

The Fabric ASIC provides non-blocking connectivity among the PFes that populate a single-chassis system. on a

single-chassis T Series system, each chassis contains a maximum of 8 FPCs, with each FPC supporting 2 PFes (up to

50 Gbps each), for a total of 16 PFes that communicate across the switch fabric.

The required input and output aggregate bandwidth of the switch fabric exceeds the I/o capabilities of a single Fabric

ASIC by a large margin. In a multichassis-capable T Series router, each PFe is connected to four active switch planes,

and each switch plane carries a portion of the required bandwidth. To guarantee that cells are evenly load-balanced

across the active switch planes, each PFe distributes cells equally across the switch planes on a cell-by-cell basis

rather than a packet-by-packet basis.

For further design considerations of the T Series switch fabric, see Appendix B: Switch Fabric Properties.

Switch Fabric OperationThe switch fabric implements a “fairness protocol” that, when congestion is occurring, ensures fairness across source

PFes. The process of transmitting a data cell across the switch fabric involves a request and grant protocol. The

source PFe transmits a request across the switch fabric to the destination PFe. each request for a given destination is

transmitted across a different switch plane in a round-robin order to distribute the load equally.

when the request is received by the destination PFe, the destination PFe transmits a grant to the source PFe across

the same switch plane on which the corresponding request was received.

when the grant is received by the source PFe, the source PFe transmits the data cell to the destination PFe across the

same switch plane on which the corresponding grant was received.

This approach provides both a flow control mechanism for transmitting cells into the fabric and a mechanism to detect

broken paths across the switch fabric.

Appendix B covers Switch Fabric Design Properties. In particular, there are two key features of the T Series switch

fabric that deserve special attention in terms of their ability to handle modern multiservice applications such as video

delivery—these are QoS and multicast.

As shown in Figure 6, any crossbar endpoint (PFe) can theoretically become congested in the egress direction:

multiple ingress PFes can send traffic to one egress PFe, thus creating potential for congestion. This condition can be

handled in any number of ways—backpressure, rate limiting, fabric speedup, or a combination—the key is that it must

not result in the loss of priority traffic.

Plane E (Backup)

Plane D

Plane C

Plane B

Plane A

Network

Network

FabricASIC

PICSI (Fabric)

PIC

EgressPFE

Network

Network

PICSI (Fabric)

PIC

IngressPFE



Multicast traffic always must coexist with unicast traffic, which is why a T Series switch fabric treats both traffic types

equally: unicast and multicast traffic must conform to their QoS profiles mapped into fabric with two priority levels

(note PFe 0).

Figure 6. Multicast and unicast each utilizing fabric queues in T Series routers

Because the T Series Core routers use a patented tree replication algorithm, they are able to distribute multicast

replication across available PFes. Thus, in the very rare case of a congested egress PFe, T Series routers can easily

apply backpressure notification from egress to ingress to limit the traffic and avoid drops. As competing designs

replicate in the fabric itself, and allow multicast traffic to bypass the fabric queues, the potential for intelligent drops

with even minimal congestion is very demonstrable.

Multichassis Switch Fabric (TX Matrix and TX Matrix Plus) An increasingly popular approach to augmenting forwarding performance, boosting bandwidth density, and extending the

deployable lifetime of core routers is to use a multichassis system. Multichassis systems are designed with an expandable

switch fabric that allows service providers to grow systems in increments required by budget and traffic loads.

while most service providers typically follow the technology curve and upgrade as soon as the next generation of

routers comes along (mainly because of improved efficiencies such as higher capacity, better footprint, and lower

power), Juniper’s multichassis solution allows providers to grow node capacity either to bridge between generations, or

to build multi-terabit nodes.

Another common reason for a service provider to use a multichassis system is to prevent the proliferation of too many

interconnects. More network elements mean more interconnects that do no support revenue-generating traffic but

merely ensure a full mesh. Interconnects are not a good way to scale—in addition to the obvious cost of the ports, there

is also a cost in terms of driving additional chassis as well to handle overhead connections.

Additionally, multichassis systems can also be combined with an independent control plane such as the JCS1200 to

create a virtualized core.3 The JCS1200 allows the creation of hardware-virtualized routers that can assume individual

networking functions, such as core or aggregation; service functions (such as private vPNs); mobile packet backbones;

or peering functions.

CLOS TopologyFigure 7 illustrates the topology of a typical three-stage CLoS fabric4. The blue squares represent individual single-

stage crossbar switches. The topology consists of multiple rows of single-stage crossbar switches arranged into three

columns. The benefit of this topology is that it facilitates the construction of large, scalable, and non-blocking switch

fabrics using smaller switch fabrics as the fundamental building block.

SWITCHFABRIC

0PFE

Routing Engine

1WAN PFE

2WAN PFE

3WAN PFE

4WANPFE

5WANPFE

6WANPFE

NWANPFE

Regular Fabric Queues(Per Destination PFE)Multicast

Input

UnicastInput



Figure 7. Three-stage CLOS topology

The topology of a CLoS network has each output port of the first stage connected to one of the crossbars in the

second stage. observe that a two-stage switch fabric would provide any-to-any connectivity (any ingress port to the

fabric can communicate with any egress port from the fabric), but the path through the switch is blocking. The addition

of the third stage creates a non-blocking topology by creating a significant number of redundant paths. It is the

presence of the redundant paths that provides the non-blocking behavior of a CLoS fabric.

Juniper Networks ImplementationThe Juniper Networks implementation of a CLoS fabric for a multichassis routing node is specifically designed to

support the following attributes:

• Non-blocking

• Fair bandwidth allocation

• Maintains packet order

• Low latency for high-priority traffic

• Distributed control

• redundancy and graceful degradation

The existence of multiple parallel paths through the switching fabric to any egress port gives the switch fabric its

rearrangeably non-blocking behavior. Dividing packets into cells and then distributing them across the CLoS switch

fabric achieves the same effect as moving (rearranging) connections in a circuit-switched network because all of the

paths are used simultaneously. Thus, a CLoS fabric proves to be non-blocking for packet or cell traffic. This design

allows PFes to continue sending new traffic into the fabric and, as long as there are no inherent conflicts—such as an

overcommitted output port—the switch remains non-blocking.

Finally, an important property of a CLoS fabric is that each crossbar switch is totally independent of the other crossbar

switches in the fabric. hence, this fabric does not require a centralized scheduler to coordinate the actions of the

individual crossbars. each crossbar acts independently of the others, which results in a switch fabric that is highly

resilient to failures and scales to support the construction of extremely large fabrics.

Multichassis System FabricThe T Series router multichassis fabric is constructed using a crossbar designed by Juniper Networks called the Fabric

ASIC, which is the same Fabric ASIC used in a single-chassis T Series platform. Depending on its placement in the

CLoS topology, each Fabric ASIC is a general building block that can perform stage 1, stage 2, or stage 3 functionality

within the switch fabric.

The CLoS fabric provides the interconnection among the PFes in a T Series multichassis routing node (Figure 7). The

switch fabric for a multichassis system is implemented using four operationally independent, but identical, switch

planes (labeled A through D) that are simultaneously active and an identical fifth plane (labeled e) that acts as a hot

spare to provide redundancy. each plane contains a three-stage CLoS fabric built using Juniper Networks Fabric ASICs.

P1

Input1

SWITCH 1

STAGE 1 STAGE 2 STAGE 3

16

Output

•••

•••

P1

•••

•••

•••

1

SWITCH 216

•••

•••

P2

Input Output

•••

•••

P2•••

•••

P3

Input Output

•••

•••

P3•••

•••

P1

1

SWITCH 1616

•••

•••

P2•••

•••

P3•••

•••



Figure 8. T Series multichassis switch fabric planes

The required input and output bandwidth of the switch fabric exceeds the I/o capabilities of a single plane. As with the

single-chassis system, each PFe is connected to four active switch planes, with each switch plane providing a portion

of the required bandwidth. To guarantee that cells are evenly load-balanced across the active switch planes, each PFe

distributes cells equally across the four switch planes on a cell-by-cell basis rather than a packet-by-packet basis.

Figure 9. Multichassis system high-level overview

Line-card chassis are connected to switch chassis with fiber-optic cable. This connectivity uses the latest vCSeL

technology to provide extremely high throughput, low power consumption, and low bit-error rates.

An abstract view of the interconnections for a TX Matrix Plus multichassis system is shown in Figure 10.

Plane E

Plane D

Plane C

Plane B

Plane A

Network

Network

PICSI (Fabric)

PIC

EgressPFE

Network

Network

PICSI (Fabric)

PIC

IngressPFE

CHASSIS 1 CHASSIS 2

F1

F1•••

F1

F2

F2•••

F2

F3

F3•••

F3

PIC

FPC

PIC

FPC

PIC

FPC

RedundantRouting Engine

Routing Engine:Local Routing Engine

PIC

FPC

PIC

FPC

PIC

FPC

RedundantRouting Engine

Routing Engine:Local Routing Engine

Matrix Control PathsMatrix Data Paths

Routing EngineSecondary

Routing EnginePrimary

T Series LCCT Series LCC

TX Matrix (Plus)



Figure 10. Switch-card chassis to line-card chassis interconnections

In all T Series systems, there is a fifth data plane on hot standby in the event any of the other four data planes fail—the

same is true for a TX Matrix Plus system.

redundancy in the center stage is provided, as the system functions with any two of the center stage chassis. each

center stage chassis has no more than two data planes, so even if one chassis fails, forwarding still proceeds at full

throughput for large packets. Five chassis can be supported by the architecture.

Conclusion The T Series demonstrates how Juniper has evolved its router architecture to achieve substantial technology

breakthroughs in packet forwarding performance, bandwidth density, IP service delivery, and system reliability. At the

same time, the integrity of the original design has made these breakthroughs possible.

Not only do T Series platforms deliver industry-leading scalability, they do so while maintaining feature and software

continuity across all routing platforms. whether deploying a single-chassis or multichassis system, service providers

can be assured that the T Series satisfies all networking requirements.

About Juniper NetworksJuniper Networks is in the business of network innovation. From devices to data centers, from consumers to cloud

providers, Juniper Networks delivers the software, silicon and systems that transform the experience and economics

of networking. The company serves customers and partners worldwide. Additional information can be found at

www.juniper.net.

F2

SIB

F2

SIB

F2

SIB

F2

SIB

F2

SIB

Blank

Blank

Blank

Blank

OpticalData Plane

TX MATRIXSwitch-Card

Chassis

RE/CB 0Standby

RE/CB 1Main

LCC 00 F13 SIB LCC 01






Blank



Blank



T1600 Plane 4

T1600 Plane 3

T1600 Plane 2

T1600 Plane 1

T1600 Plane 0 - Standby

T1600Line-Card

Chassis

RE/CB 1Main

RE/CB 0Standby



Appendix A: References

Applications for an Independent Control Plane:

www.juniper.net/us/en/local/pdf/app-notes/3500134-en.pdf

Control Plane Scaling and router virtualization:

www.juniper.net/us/en/local/pdf/whitepapers/2000261-en.pdf

efficient Scaling for Multiservice Networks:


energy efficiency for Network equipment:


Network operating System evolution:


virtualization in the Core of the Network:




Appendix B: Switch Fabric PropertiesThe T Series switch fabric for both single-chassis and a multichassis system is specifically designed to provide the

following attributes.

• Non-blocking

• Fair bandwidth allocation

• Maintains packet order

• Low latency for high-priority traffic

• Distributed control

• redundancy and graceful degradation

Non-blockingA switch fabric is considered non-blocking if two traffic flows directed to two different output ports never conflict. In

other words, the internal connections within the switch allow any ingress PFe to send its fair share of bandwidth to any

egress PFe simultaneously.

Figure 11. A 4 x 4 non-blocking crossbar switch

Figure 11 illustrates the internal topology for a non-blocking, single-stage, 4-port crossbar switch. The challenge

when building a crossbar is that it requires n² communication paths internal to the switch. In this example, the

4-port crossbar requires a communication path connecting each input port to each output port, for a total of 16

communication paths. As the number of ports supported by the crossbar increases, the n² communication path

requirement becomes an implementation challenge.

Fair Bandwidth AllocationIn a production network, it is impossible to control the pattern of ingress traffic so that an egress port of the crossbar

switch is never overcommitted. An egress port becomes overcommitted when there is more input traffic destined for

the egress port than the egress port can forward. In Figure 12, the aggregate amount of traffic that ingress ports 1, 2, and

3 forward to egress port 4 is greater than the capacity of egress port 4. Fair bandwidth allocation is concerned with the

techniques that a switch uses to share the bandwidth among competing ingress flows to an overcommitted egress port.

Figure 12. An overcommitted egress switch port

The T Series router switch fabric provides fairness by ensuring that all ingress PFes receive an equal amount of

bandwidth across the switch fabric when transmitting cells to an oversubscribed egress PFe. Providing this type of

fairness across all streams is hard to support because it is difficult to keep track of all users of the bandwidth to the

egress PFe. The challenge is that if there are n ports on the switch, then there are n² streams of traffic through the

Port 1 Ingress Port 1 Egress

4X4 CROSSBAR SWITCH





4X4 CROSSBAR SWITCH






2000302-003-eN Mar 2012

Copyright 2012 Juniper Networks, Inc. All rights reserved. Juniper Networks, the Juniper Networks logo, Junos, NetScreen, and ScreenoS are registered trademarks of Juniper Networks, Inc. in the united States and other countries. All other trademarks, service marks, registered marks, or registered service marks are the property of their respective owners. Juniper Networks assumes no responsibility for any inaccuracies in this document. Juniper Networks reserves the right to change, modify, transfer, or otherwise revise this publication without notice.

EMEA Headquarters

Juniper Networks Ireland

Airside Business Park

Swords, County Dublin, Ireland

Phone: 35.31.8903.600

eMeA Sales: 00800.4586.4737

Fax: 35.31.8903.601

APAC Headquarters

Juniper Networks (hong Kong)

26/F, Cityplaza one

1111 King’s road

Taikoo Shing, hong Kong

Phone: 852.2332.3636

Fax: 852.2574.7803

Corporate and Sales Headquarters

Juniper Networks, Inc.

1194 North Mathilda Avenue

Sunnyvale, CA 94089 uSA

Phone: 888.JuNIPer (888.586.4737)

or 408.745.2000

Fax: 408.745.2100

www.juniper.net

To purchase Juniper Networks solutions,

please contact your Juniper Networks

representative at 1-866-298-6428 or

authorized reseller.

Printed on recycled paper

switch. Since most switch architectures are not capable of keeping track of n² individual streams, they are forced

to aggregate traffic streams, thus making it impossible to be completely fair to each individual stream. The T Series

switch fabric can monitor the n² streams so that each stream receives its fair share of the available fabric bandwidth to

an oversubscribed egress PFe.

Maintains Packet OrderThe potential for misordering cells as they are transmitted across parallel switch planes to the egress PFe is eliminated

by the use of sequence numbers and a reorder buffer. In this design, the Switch Interface ASIC on the ingress PFe places a

sequence number into the cell header of each cell that it forwards into the fabric. on the egress PFe, the Switch Interface

ASIC buffers all cells that have sequence numbers greater than the next sequence number it waits to receive.

If a cell arrives out of order, the Switch Interface ASIC buffers the cells until the correct in-order cell arrives and the

reorder buffer is flushed. The reorder buffer and sequence number space are large enough to ensure that packets (and

the cells in a given packet) are not reordered as they traverse the switch fabric.

Low Latency for High-Priority TrafficSome types of traffic, such as voice or video, have both latency and bandwidth requirements. The T Series switch fabric

is designed so that blocking in the fabric is extremely rare because the size of the ports from the ingress PFe into the

fabric is considerably larger than the network ports into the ingress PFe.

In the rare case that congestion does occur, each ingress PFe is allocated priority queues into the switch fabric. As

previously discussed, the switch fabric fairly allocates bandwidth among all of the ingress PFes that are competing to

transmit cells to an overcommitted egress PFe.

• The use of priority queues from the ingress PFe into the switch fabric provides two important benefits:

• The latency for high-priority traffic is always low because the fabric never becomes congested.

• The CoS intelligence required to perform admission control into the fabric is implemented in the PFes. This design allows

the switch fabric to remain relatively simple because CoS is not implemented inside the fabric, but at the edges of the

switch fabric. It also scales better than a centralized design because it is distributed and grows with the number of FPCs.

Distributed ControlA T Series routing platform does not have a centralized controller that is connected to all of the components in the switch

fabric. hence, within the fabric, if any component fails, the other components around the failed component continue to

operate. Additionally, a centralized control channel does not need to be operational for the switch fabric to function.

Redundancy and Graceful Degradationeach Switch Interface ASIC monitors a request-grant mechanism. If the ingress PFe depends on a grant for an

outstanding request but the grant does not return after a reasonable amount of time, or a data cell is lost, then the

ingress PFe takes into account that the destination PFe is unreachable on the plane that was used to send the request.

If a switch plane fails, only the cells that are currently in transit across the switch plane are lost because buffering does

not occur within a switch plane and the request-grant mechanism ensures that cells are never transmitted across a

failed switch plane.

The request-grant mechanism allows a failed component within a switch plane to be removed from service by diverting

traffic around the faulty component, or all traffic using the faulty plane can be switched to a redundant plane. If there

are a significant number of faults on a given switch plane or the plane must be swapped out for maintenance, the

chassis manager coordinates moving traffic to the redundant switch plane. each step in the migration of traffic to

the redundant plane involves moving only a small fraction of overall traffic. This design allows the system to remain

operational with no significant loss in fabric performance.

http://www.juniper.net

Technology

T Series Core Router Architecture Review (Whitepaper)