Chiaro’s Enstara™ Summary

• Scalable Capacity– 6 Tb/S Initial Capacity– GigE OC-192 Interfaces– “Soft” Forwarding Plane With Network Processors For

Maximum Flexibility• Full protocol suite

– Unicast: BGP, OSPF, IS-IS– Multicast: PIM, MBGP, MSDP– MPLS: RSVP-TE, LDP, FRR

• Stateful Assured Routing (STAR™) – Provides Service Continuity During Maintenance and

Fault Management Actions– Stateful Protocol Protection Extended to BGP, ISIS,

OSPF, Multicast, and MPLS• Partitions

– Abstraction Permitting Multiple Logical Classes Of Routers To Be Managed As If Separate Physical Routers

– Each Partition Has Its Own CLI, SNMP, Security, and Routing Protocols Instances

We Chose OptIPuter for Fast Switching and Scalability

Lithium Lithium

NiobateNiobate

ElectricalElectrical

FabricsFabricsBubbleBubble

Packet

Switching

Speeds

(ns)

Switching

Speeds

(ms)

Small

Port

Count

Large

Port

Count

ChiaroChiaro

OpticalOptical

PhasedPhased

ArrayArray

MEMSMEMS

ElectricalElectrical

FabricsFabrics

InputOptical Fiber

WG #1 WG #128

• • • • • •

Optical Phased Array –Multiple Parallel Optical Waveguides

OutputOutputFibersFibers

Air G

ap

Air G

ap

GaAs Waveguides

Chiaro Has a Scalable, Fully Fault Tolerant Architecture

• Significant Technical Innovation– OPA Fabric Enables

Large Port Count– Global Arbitration

Provides Guaranteed Performance

– Fault-Tolerant Control System Provides Non-stop Performance

• Smart Line Cards – ASICs With

Programmable Network Processors

– Software Downloads For Features And Standards Evolution

NetworkNetworkProc.Proc.LineLineCardCard

NetworkNetworkProc.Proc.LineLineCardCard

GlobalGlobalArbitrationArbitration

Optical Electrical

ChiaroOPA

Fabric

NetworkNetworkProc.Proc.LineLineCardCard

NetworkNetworkProc.Proc.Line Line CardCard

• Cluster – Disk

• Disk – Disk

• Viz – Disk

• DB – Cluster

• Cluster – Cluster

OptIPuter LambdaGridEnabled by Chiaro Networking Router

www.calit2.net/news/2002/11-18-chiaro.html

switch switch

switchswitch

Medical Imaging and Microscopy

Chemistry, Engineering, Arts

San Diego Supercomputer Center

Scripps Institution of Oceanography

ChiaroEnstara

Image Source: Phil Papadopoulos, SDSC

½ Mile

The UCSD OptIPuter Deployment

SIO

SDSC

CRCA

Phys. Sci -Keck

SOM

JSOE Preuss

6th College

Phase I, Fall-Winter 2002

Phase II, 2003

SDSCAnnex

Collocation point

Node M

The OptIPuter Experimental UCSD Campus Optical Network

Earth Sciences

SDSC

Arts

Chemistry

Medicine

Engineering

High School

UndergradCollege

SDSCAnnex

To CENIC

Collocation

Chiaro Router (Installed Nov 18, 2002)

Production Router (Planned)

Source: Phil Papadopoulos, SDSC; Greg Hidley, Cal-(IT)2

Roughly, $0.20 / Strand-Foot

UCSD New Cost Sharing Roughly $250k of Dedicated Fiber

UCSD Chiaro Estara

• Full Layer-3 Router Operating at 10ns– Chiaro Estara Router Chassis

– Installed November 18, 2002

– 24 1 GigE Network Interfaces– Spring 2003

• Expandable to:– 320 x 10 GigE Network Interfaces = 6.4 Terabit/s

– All Capable Or Concurrent Routing At Full Line Speed

• Webcam– http://132.239.26.190/view/view.shtml

The Center of the Network

Status as of about 15 minutes ago

• The hardest thing in networking – getting fiber in the ground and terminated– 4 pair single mode fiber per site– Fiber terminated Monday (3 days ago) – fiber “polishing” still ongoing – First light/first packet yesterday (wed) at about 6pm– Ran linpack at about 6:05pm

• Currently two sites are connected– Linux Cluster at each site as a baseline. Sizes/capability/architecture vary as needed

(and $$) change. – 4 x 1 gigE @ CSC– 4 x 1 gigE @ SDSC

• Chiaro– Serial #1 production router. “Tiny size”:

– One redundant pair of optical cores – 640 gigabits. Can grow to 6+ Terabits– We have alpha/beta gigE blades from Chiaro

– OptIPuter getting these 3 – 4 months ahead of schedule– Chose multiple gigE striped physically because

– Cost at the site end– Start parallel

– From the endpoint view – Chiaro works just like a standard router.

Netperf Numbers (data at first light)

SDSC -> CSC netperf( 2 X 1GHz PI I I --> 2 x 2.2 GHz PIV)

0

200

400

600

800

1000

1200

1400

1600

1800

0 10000 20000 30000 40000 50000 60000 70000

Message Size (bytes)

Band

wid

th (

Mb/

s) Stream 1Stream 2Aggregate

• 2 streams more than 1 gigabit (aggregate)

Linpack Numbers

• 4 processor Linpack– Run through local copper gigE switch– Run through Chiaro Router

Linpack (4 CPUs/4 Nodes)

0

100

200

300

400

500

600

700

1000 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000

Matirx Dimension

Tim

e (s

ec)

0

1

2

3

4

5

6

7

8

9

10

Gig

aFlo

ps Local GigE sec

Chiaro sec

Local GigE Gflops

Chiaro Gflops

(17.6 Gflops Peak -- Four 2.2GHz Pentium 4)