6 May 2002

Next Generation Abilene and Advanced National & Regional InfrastructureNext Generation Abilene and Advanced National & Regional Infrastructure

CENIC 2002

San Diego

CENIC 2002

San Diego

6 May 2002 2

Key topics

Abilene Network upgrade

Quilt Project

Optical networking progress

6 May 2002 3

Abilene background & milestones

Abilene is a UCAID project in partnership with• Qwest Communications (SONET & DWDM service)• Cisco Systems (routers, switches & access)• Juniper Networks (routers)• Nortel Networks (SONET kit)• Indiana University (network operations)• ITECs in North Carolina and Ohio (test and evaluation)

Timeline• Apr 1998: Project announced at White House• Jan 1999: Production status for network• Oct 1999: IP version of HDTV (215 Mbps) over Abilene• Nov 2001: Raw HDTV/IP (1.5 Gbps) over Abilene

6 May 2002 4

Abilene – May, 2002

IP-over-SONET backbone (OC-48c, 2.5 Gbps) 53 direct connections

• 4 OC-48c connections • 1 Gigabit Ethernet trial • 23 will connect via at least OC-12c (622 Mbps) by 1Q02• Number of ATM connections decreasing

215 participants – research universities & labs• All 50 states, District of Columbia, & Puerto Rico• 15 regional GigaPoPs support ~70% of participants

Expanded access• 50 sponsored participants

– New: Smithsonian Institution, Arecibo Radio Telescope

• 23 state education networks (SEGPs)

6 May 2002 5

6 May 2002 6

Abilene international connectivity

Transoceanic R&E bandwidths growing!• GÉANT – 5 Gbps between Europe and New York City now

Key international exchange points facilitated by Internet2 membership and the U.S. scientific community

• STARTAP & STAR LIGHT – Chicago (GigE)• AMPATH – Miami (OC-3c OC-12c)• Pacific Wave – Seattle (GigE)• MAN LAN - New York City (GigE/10GigE EP soon)• CA*NET3/4: Seattle, Chicago, and New York• CUDI: CENIC and Univ. of Texas at El Paso

International transit service• Collaboration with CA*NET3 and STARTAP

6 May 2002 7

Packetized raw High Definition Television (HDTV)

Raw HDTV/IP – single UDP flow of 1.5 Gbps• Project of USC/ISIe, Tektronix, & U. of Wash (DARPA) • 6 Jan 2002: Seattle to Washington DC via Abilene

–Single flow utilized 60% of backbone bandwidth• 18 hours: no packets lost, 15 resequencing episodes• End-to-end network performance (includes P/NW & MAX

GigaPoPs)– Loss: <0.8 ppb (90% c.l.)– Reordering: 5 ppb

• Transcontinental 1-Gbps TCP requires loss of

– <30 ppb (1.5 KB frames)– <1 ppm (9KB jumbo)

6 May 2002 8

End-to-End Performance:‘High bandwidth is not enough’

Bulk TCP flows (transfers > 10 Mbytes) • Current median flow rate over Abilene: 1.9 Mbps

–95th percentile: 7.0 Mbps

6 May 2002 9

Future of Abilene

Original UCAID/Qwest agreement amended on October 1, 2001

Extension of for another 5 years – until October, 2006

• Originally expired March, 2003

Upgrade of Abilene backbone to optical transport capability - ’s (unprotected)

• x4 increase in the core backbone bandwidth–OC-48c SONET (2.5 Gbps) to 10-Gbps DWDM

6 May 2002 10

Key aspects of next generation Abilene backbone - I

Native IPv6• Motivations

– Resolving IPv4 address exhaustion issues

– Preservation of the original End-to-End Architecture model• p2p collaboration tools, reverse trend to CO-centrism

– International collaboration

– Router and host OS capabilities

• Run natively - concurrent with IPv4• Replicate multicast deployment strategy• Close collaboration with Internet2 IPv6 Working Group on

regional and campus v6 rollout– Addressing architecture

6 May 2002 11

Key aspects of next generation Abilene backbone - II

Addition of new measurement capabilities• Enhance active probing (Surveyor)

– Latency & jitter, loss, TCP throughput

• Add passive measurement• Support for computer science research – “Abilene Observatories”

– archive/measurement/experiment• Support of Internet2 End-to-End Performance Initiative

– Intermediate performance beacons

Network resiliency• Abilene ’s will not be protected like SONET• Increasing use of videoconferencing/VoIP impose tighter

restoration requirements (<100 ms)• Options:

– MPLS/TE fast reroute (initially)– IP-based IGP fast convergence (preferable)

6 May 2002 13

Next generation router selection

Extensive router specification and test plan developed

• Team effort: UCAID staff, NOC, NC and Ohio ITECs• Discussions with four router vendors

Tests focused on next gen advanced services• High performance TCP/IP throughput • High performance multicast• IPv6 functionality & throughput• Classification for QoS and measurement

3 router platforms tested & commercial ISPs referenced

New Juniper T640 platform selected

6 May 2002 14

Two leading national initiatives in the U.S.

Next Generation Abilene• Advanced Internet backbone

– connects entire campus networks of the research universities

• 10 Gbps nationally

TeraGrid• Virtual machine room for distributed computing (Grid) • Connecting 4 HPC centers initiallly

– Illinois: NCSA, Argonne– California: SDSC, Caltech

• 4x10 Gbps: Chicago Los Angeles

Ongoing collaboration between both projects

6 May 2002 15

Quilt Project

Coalition of advanced regional networks • Now includes 18 GigaPoPs with SURA and EDUCAUSE• Including CENIC and Pacific/Northwest • Led by Wendy Huntoon (Pittsburgh)

Became a project of UCAID last summer• In parallel with Internet2 & Abilene projects

Core activities• Commercial Internet Services• Regional Optical Networking ( Next Gen Abilene)• Measurement ( End-to-end Performance)• New Internet2/Quilt K-20 Joint Initiative

6 May 2002 16

Optical networking technology drivers

Aggressive period of fiber construction on the national & metro scales in U.S.

Many university campuses and regional GigaPoPs with dark fiber

Dense Wave Division Multiplexing (DWDM)

• Allows the provisioning of multiple channels (’s) over distinct wavelengths on the same fiber pair

• Fiber pair can carry 160 channels (1.6 Tbps!)

Optical transport is the current focus• Optical switching is still in the realm of experimental

networks, but is nearing practical application

6 May 2002 17

DWDM technology

DWDM fundamentally is an analog optical technology

• Combines multiple channels (2-160+) over the same fiber pair

• Employs discrete wavelengths (’s) of light • Generally supports 2.5 or 10 Gbps channels

Physical obstacles to long-distance transmission of light

• Attenuation –Solved by amplification (OO)

• Dispersion–Requires periodic signal regeneration (OEO)

18

Optical network project differentiation

Distance scale (km)

Examples Equipment

Metro < 60

UW(SEA),

USC/ISI(LA)

Dark fiber & end terminals

State/

Regional

< 500

(ULH: <2500)

I-WIRE (IL),

CENIC ONI,

I-LIGHT (IN)

Add OO

amplifiers

Extended

Regional/

National

> 500

PLR,

TeraGrid

Abilene

Add OEO

regenerators

& O&M $’s

6 May 2002 19

Regional optical fanout

Next generation architecture: Regional & state based optical networking projects are critical

• Three-level hierarchy: Backbones, GigaPoPs, campuses• Leading examples

– CENIC ONI (California), I-WIRE (Illinois), I-LIGHT (Indiana)

• State-based projects can be viewed as cost sharing

Collaboration with the Quilt • Regional Optical Networking project • Interface with next generation Abilene and future projects

U.S. carrier DWDM access is not widespread• 30-60 cities for DWDM now• ~120 cities for SONET (circa 1998)• Further expansion constrained by current telecom crunch

6 May 2002 20

National optical networking options

1 – Provision incremental wavelengths• Obtain 10-Gbps ’s as with SONET• Exploit smaller incremental cost of additional ’s

– 1st cost is ~10x than subsequent ’s

2 – Build dim fiber facility• Partner with a facilities-based provider

– Acquire 2 fiber pairs on a national scale– Outsource operation of transmission equipment

• Needs lower-cost optical transmission equipment– Find ELH/ULH optical kit partner

The classic ‘buy vs. build’ decision in Information Technology

• Option 1 selected for TeraGrid and Next Gen Abilene

6 May 2002 21

Conclusions – Abilene future

Backbone upgrade project underway• Partnership with Qwest extended thru 2006• Juniper T640 routers selected for backbone• 10-Gbps backbone deployment starts this fall

Advanced service foci• Native, high-performance IPv6• Enhanced, differentiated measurement• Network resiliency

Incremental, non-disruptive transition

Complementary to and collaborative with NSF’s TeraGrid

www.internet2.edu