Internet2: Supporting Next Generation Networking

Christian Todorov

August 16, 2010

NASA IT Summit 2010

Session Agenda

• Internet2 Overview• Network Architecture • Network Services• IP• Circuit Service• Use of Dynamic Circuits• Waves and Private Networking

• Supporting Advanced Research• Discussion

INTERNET2 OVERVIEW

The Internet2 Network Overview

• Layer 1: Managed wavelengths from Level(3) Communications • Level(3) owns and manages Infinera optical gear: responsible for

software upgrades, equipment maintenance, remote hands, sparing, NOC services

• Internet2 NOC has total provisioning control• Layer 2: Internet2 owned and managed equipment

• Using OSCARS and DRAGON GMPLS control plane • Layer 3: Internet2 owned and managed Juniper MX960s • Expanded Observatory

• Platform for layer 1/3 network performance data collection, collocation, experimentation

• perfSONAR integration for intra- & inter-network performance analysis• International connectivity

• Layer 1 network extended to international exchange points in Seattle, Chicago and New York City

• Peering points in Seattle, PAIX, Equinix Chicago

4

Internet2 Connected Community

• 217 Leading US research institutions• ~50,000 K20, libraries, museums, zoos and aquariums• 38 states covered by SEGP program• Federal network peerings and International peerings

reaching 80 countries• Private networking provided to NOAA and Department of

Energy (ESnet); Piloting with USFS• Named participant in the FCC’s RHCPP; connecting rural

healthcare networks• Actively working with the VA• JPL and Goddard are Affiliate Members • Internet2 peers with NASA networks: NREN and NISN as

well as others such as DREN and USGS

5

Internet2 Universities217 University Members as of June 2010

Sample Listing of Government Affiliates

Internet2 Affiliate Members• ACUTA: The Assn. for Information

Communications Professionals in Higher Ed.

• Altarum

• Ames Laboratory, US Department of Energy

• Argonne National Laboratory

• Association of Universities for Research in Astronomy

• Berklee College of Music

• Brookhaven National Laboratory

• CineGrid, Inc.

• Coalition for NeCleveland Institute of Music

• tworked Information (CNI)

• Department of Veterans Affairs (VA)

• Desert Research Institute

• EDUCAUSE

• ESnet

• Federal Highway Administration, Turner-Fairbank Highway Research Center

• Fermi National Accelerator Laboratory

• HIMSS (Healthcare Information and Management Systems Society)

• Howard Hughes Medical Institute

• IEEAF (Internet Educational Equal Access Foundation)

• Indian Health Services

• Indiana Higher Education Telecommunication System (IHETS)

• ISOC (Internet Society)

• Jet Propulsion Lab

• JSTOR

• Laboratory for Telecommunications Sciences (LTS)

• Lawrence Berkeley National Laboratory

• Library of Congress

• Los Alamos National Laboratory

• Manhattan School of Music

• NASA Goddard Space Flight Center

• National Archives and Records Administration

• National Emergency Number Association (NENA)

Internet2 Affiliate Members• National Geographic

• National Institutes of Health

• National Science Foundation

• New World Symphony

• NIST (National Institute of Standards and Technology)

• NITLE (National Institute for Technology and Liberal Education)

• NOAA (National Oceanic & Atmospheric Administration, Washington, D.C.)

• Oak Ridge National Laboratory

• OCLC Online Computer Library Center

• Pacific Northwest National Laboratory

• Princeton Plasma Physics Laboratory

• Ruth Lilly Health Education Center

• Stanford Linear Accelerator Center

• SURA (Southeastern Universities Research Association)

• The Children's Hospital of Philadelphia

• The Philadelphia Orchestra Association

• The World Bank

• Thomas Jefferson National Accelerator

• TOPIX (Torino E Piemonte Exchange Point)

• UCAR/NCAR (University Corporation for Atmospheric Research)

• United Nations System of Organizations (UN)

• United States Antarctic Program (USAP)

• United States Department of Agriculture, Forest Service (USFS)

• United States Department of Commerce Boulder Labs

• United States Department of State

• United States Holocaust Memorial Museum

• University of North Carolina General Administration

10

The Internet2 Network

• Built on dedicated fiber from Level(3) Communications – 13,000+ mile footprint

• Currently based on 10G waves; moving to 100G• Built on Infinera innovative optical technology

• Simple and convenient add/drop technology• Simple and convenient wave setup• Demonstrated high reliability in initial period of operation on the

Level(3) network• Economics of Infinera system are disruptive in the market place

• Architecture has maximum flexibility • Every direct connector can access every wave on the system if

needed• Adding add/drop points doesn’t require network redesign

Network Design

Internet2 Network

International Connectivity

Slide 12

T640

T640

T640

T640

T640

T640

T640

T640

T640

PacWave: APAN/TransPac2, AARnet, KOREN/KREONet2, CA*Net4, GEMNET, REANNZ,

France Telecom (v6), HARNET, CERNStarLight: CERN, JGN2, ASNet, CERN, KOREN/KREONet2, CA*Net4, CERN, JGN2, SINET, GLORIAD, TANet2/TWARENNGIX-Ames:

AARnet

PacWave:UNINET, SINET, QatarFN, APAN/TransPac2 TANet2/TWAREN, SingAREN

redCLARA, CUDI via CALREN/PacWave

CUDI via UTEP / UT

AMPATH/SFGP:ANSP, REACCIUN-2, redCLARA

NGIX-East/A-Wave: CLARA, GEANT, RNP2

MANLAN:TANet2/TWAREN, TENET, MCIT/ENERGI, LHCnet, QatarFN, CA*Net4, SURFNet, CERN, GEANT(2), SINET, NetherLight/IEEAF

Europe-Middle EastAustria (ACOnet)Belgium (BELNET)Croatia (CARNet)Czech Rep. (CESNET)Cyprus (CYNET)Denmark (Forskningsnettet)Estonia (EENet)Finland (Funet)France (Renater)Germany (G-WIN)Greece (GRNET)Hungary (HUNGARNET)Iceland (RHnet)Ireland (HEAnet)Israel (IUCC)Italy (GARR)Latvia (LATNET)Lithuania (LITNET)

Asia-Pacific AmericasArgentina (RETINA)Brazil (RNP2/ANSP)Canada (CA*net)Chile (REUNA)Mexico (Red-CUDI)United States (Abilene)Peru (RAAP)Venezuela (REACCIUN-2)

Last updated: April 2005

M Luxembourg

(RESTENA) alta (Univ. Malta)Netherlands (SURFnet) Norway (UNINETT)Poland (POL34)Portugal (RCTS2)Qatar (Qatar FN)Romania (RoEduNet)Russia (RBnet)Slovakia (SANET)Slovenia (ARNES)Spain (RedIRIS)Sweden (SUNET)Switzerland (SWITCH)United Kingdom (JANET)Turkey (ULAKBYM)*CERN

Australia (AARNET)China (CERNET, CSTNET, NSFCNET)Hong Kong (HARNET)Japan (SINET, WIDE, JGN2)Korea (KOREN, KREONET2)Singapore (SingAREN)Philippines (PREGINET)Taiwan (TANet2, ASNet)Thailand (UNINET, ThaiSARN)

Algeria (CERIST)Egypt (EUN/ENSTIN)Morocco (CNRST)Tunisia (RFR)South Africa (TENET)

Central Asia AfricaArmenia (ARENA)Georgia (GRENA)Kazakhstan (KAZRENA)Tajikistan (TARENA)Uzbekistan (UZSCI)

Internet2 Network

International Connectivity

Slide 13

Internet2 Network

Federal Peering

Slide 14

T640

T640

T640

T640

T640

T640

T640

T640

T640

PacWave:DREN, NREN, ESnet

StarLight:TeraGrid, NREN, DREN, NISN, USGS, ESNet

NGIX-Ames: NREN, DREN, NISN, USGS

DC: DRAGONNGIX:DREN, USGS, NISN, NREN, Wright Patt AFB, ESNet

New York: ESnet

Network Architecture

Objectives

• The vision for the Internet2 Network is a seamless integrated network facility that allows for applications and users to transparently utilize the services and network layers that most appropriately serve their needs, when they need it, in a cost effective manner.

• This network facility will allow users to focus on their work and not on the network.

Scalability and Operability

• The Internet2 Network is based on a unique arrangement with Level 3 that represents a hybrid approach to carrier provided resources.

• Internet2 has full control over the provisioning on the network but does not bear the responsibility of supporting and maintaining the physical infrastructure: fiber, amps, transport equipment, etc.

• Level 3’s support of the physical network frees Internet2 of having higher levels of specialized engineering resources dedicated to network support.

• The Internet2 NOC has a full view into the underlying transport equipment and works jointly with a dedicated NOC group within Level 3.

• The Internet2 network is constructed on a dedicated fiber pair and with dedicated transport equipment

• The Infinera, and Juniper equipment used in the network are 40G capable and each has 100G on their roadmaps

17

Collocation Facilities

• 30 facilities across the US• 27x Level 3• 1x Westin Building, Seattle• 1x 32 AoA, New York• 1x 710 N. Lakeshore Drive, Chicago

• Dedicated suites or colo areas – secured• Supports Internet2, ESnet, NOAA and

some Internet2 connector equipment

Optical Network

• Infinera optical transmission equipment• Managed by Level 3 Communications• Internet2 has direct provisioning control• Internet2 has direct TL-1 monitoring

• Infinera nodes• Redundant Add / Drop (28)• Regeneration (28)• Optical Amplification (193)

Internet2 Network

IP Network Services• Carrier-class IP service

• Natively supports IPv4 and IPv6, as well as multicast and jumbo frames

• Carried over 10Gbps waves on the extremely reliable Infinera backbone• 40Gbps and 100Gbps are future enhancements

• Backhaul to the nearest router is included in the IP connection • Redundancy options

• One option is a geographically diverse backup connection. This connection is viewed as a backup and the aggregate traffic flow over the primary and redundant circuit must not exceed that of the primary circuit

• Internet2 currently supports 2 x 5 Gig and 2 x 10 Gig connections as well. These can be in geographically diverse locations.

• Protection against fiber cuts using the automatic restoration capabilities of the Juniper MX’s MPLS Fast Reroute as an optional service

• Additional IP circuits are also available

Slide 20

Internet2 Network

WaveCo Static Circuit Services

• Are currently set up manually by the Internet2 NOC.• Can have any bandwidth up to a full 10Gbps wave or multiple 10G

waves • Can be provisioned directly over the Infineras (waves) or through

the Junipers (sub-rate circuits)• This provisioning can occur over the same physical interface that is used for

the Dynamic Circuit Network

• Circuits can be offered both as protected and unprotected• Can be ordered for a period of weeks up to years • Have a fee for service that is based on distance, bandwidth,

duration and protection scheme.

Slide 21

Internet2 Network

Dynamic Circuit Network• The dynamic circuit network (ION) automatically provisions circuits

among participants deploying the dynamic networking control plane and protocol

• Each participant may connect to one or more circuits allowing them to customize the infrastructure to meet their application needs

• The ION service is currently based on connections into the Juniper MX960s.

• As the ION service grows and other R&E networks across the world “peer” with the ION Network, through exchange points, world wide reachability will grow.

Slide 22

ION Service• ION has moved from a SONET-based network on the Ciena’s to an MPLS-based service operating on

the current IP network.• This transition took place during the summer

• ION connectivity and provisioning options for Internet2 Network Participants will increase. • A Connector need only maintain a single connection that can provide both IP and ION services on the same

port. The Connector is empowered to appropriate resources in a way that works with their regional model.

• The shared MPLS model can increase the efficiency of the backbone. • Bandwidth reserved for a circuit instantiation is available for use by other network users when the circuit owner

is not utilizing the circuit for transfer• There’s opportunity to provide circuits that can burst above their requested commit rate, if there is sufficient

headroom available.

• ION is a production service managed by the IU NOC.

• ION circuits are provisioned using a simple and secure web based interface.

• ION is available to any institution regardless of their membership status.

• The goal is to make the service available to the entire connector community.

• Connectors or sites that use ION should contact us to discuss the best way to utilize this new model.• Contact Dale Finkelson (dmf@internet2.edu) for further information.

I2 DCN Software Suite

• OSCARS (IDC)• Originally an ESnet Project, now joint development by

Internet2 and ESnet • Open source project maintained by Internet2 and

ESNet

• DRAGON (DC)• NSF-funded• Open source project maintained by USC ISI EAST and

MAX

InterDomain Controller (IDC) Protocol (IDCP)

• Developed via collaboration with multiple organizations• Internet2, ESnet, GEANT2, Nortel, University of Amsterdam, others

• The following organizations have implemented/deployed systems which are compatible with this IDCP• Internet2 Dynamic Circuit Network (ION)• ESNet Science Data Network (SDN)• GÉANT2 AutoBahn System• Ciena (Nortel) (via a wrapper on top of their commercial DRAC System)• Surfnet (via use of above Nortel solution)• LHCNet (use of I2 ION Software Suite)• Nysernet (use of I2 ION Software Suite)• University of Amsterdam (use of I2 ION Software Suite)• DRAGON Network

• The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP:• LambdaStation (FermiLab)• TeraPaths (Brookhaven)• Phoebus

ION Control Plane

Domain Controller

Network 1

IDC

Domain Controller

Network 2

IDC

User Request/IDC Response

IDC to IDC communication

Domain Controller

Network 3

IDC

IDC to IDC communication

Multi-Service/Domain/Layer/Vendor Provisioning

Regional Network Regional

Network

Internet2 Network

ESNet

Dynamic Ethernet Dynamic EthernetTDM

GEANT

IP Network (MPLS, L2VPN)

Ethernet

Router

SONET Switch

Ctrl Element

Domain Controller

LSP

Data PlaneControl Plane Adjacency

• Multi-Domain Provisioning• Interdomain ENNI (Web Service and OIF/GMPLS)• Multi-domain, multi-stage path computation process• AAA• Scheduling TDM

Slide from Tom Lehman, ISI-East

GUI

XML

API

Application Interfaces to ION

• Phoebus is network middleware to facilitate high throughput for the non-wizard• Based on a “Session” Protocol (Layer 5) that talks to

Phoebus Gateways• These Gateways do Transport protocol translation and

tuning• Transparently allocates circuits over the Internet2 ION

• TeraPaths is a system from BNL• Creates end-to-end virtual paths and guaranteed

bandwidth for specific flows with DiffServ marking• Web interface for manual reservations and a Web

Services interface for automatic reservation• Allocates resources on the ESNet SDN or Internet2 ION

• Phoebus Gateways are placed at network access points and can be thought of as network “on-ramps

• At a high level, the goal is to get network data to these gateways which can then intelligently move the data as efficiently as possible to the other edge of the network

• Phoebus will automatically utilize ION resources as needed and as available• At the other edge, it again appears as legacy traffic does today, thus neither client

or server need to be modified

Waves and Private Networking

• Internet2 makes dedicated waves available over its network to support researchers and organizations

• Internet2 has provided nation-wide dedicated network infrastructure to support the needs of advanced research networks like ESnet (DoE) and N-Wave (NOAA) allowing for private research networking that has access to the broader Internet2 community

Monitoring and Measurement

• Dependence on the network is increasing• Distributed applications & moving larger data sets

• Network is growing much more complex• Need to better understand the network

• Transparent network operation• User must know what performance levels to expect• Network operators must be able to demonstrate that the

network meets or exceeds those expectations.• Application developers must have access to tools that

differentiate between network problems and application problems.

Internet2 Network

Observatory

Slide 32

Observatory Services

• Several tools and services are available for network troubleshooting:• Active testing to the middle to divide and conquer

• OWAMP (latency), BWCTL (throughput), NDT (last-mile/end station)

• perfSONAR (cross-domain tool results, link utilization, and test point locations)

• BGP feeds to RouteViews• Link utilization at 10 second intervals• “visible backbone” gives router configuration snapshots• “Router” proxy for interactive query

• http://www.internet2.edu/observatory/ (data views)• http://noc.net.internet2.edu/

33

perfSONAR

• A way to export measurement data from your network, and consume it from many networks (including your own)

• “Interoperable network measurement middleware”

• A set of schemas / “APIs” and protocols

35

perfSONAR Integrates

• Network measurement tools and archives (e.g. stored measurement results)

• Data manipulation• Information Services• Discovery• Topology• Authentication and authorization

36

37 – 04/19/23, © 2009 Internet2

37

FNAL (AS3152)[US]

ESnet (AS293)[US]

GEANT (AS20965)[Europe]

DFN (AS680)[Germany]

DESY (AS1754)[Germany]

measurement archive

m1m4

m3

measurement archive

m1m4

m3

measurement archive

m1m4

m3

m1m4

m3

m1m4

m3

measurement archive

measurement archive

performance GUI

user

Analysis tool

Many collaborations are inherently multi-domain, so

for an end-to-end monitoring tool to work

everyone must participate in the monitoring

infrastructure

Many collaborations are inherently multi-domain, so

for an end-to-end monitoring tool to work

everyone must participate in the monitoring

infrastructure

Meeting Upcoming Application Demands

38

Demands on the Network

• Entering the age of large scientific facilities• Large Hadron Collider at CERN• Very Long Baseline Arrays (radio astronomy)• Large Synoptic Survey Telescope (2010-13) – 30TB/night

• An increasingly diverse set of demanding applications are utilizing network resources• Telemedicine: BIRN project, proteomics, tele-surgery, remote

ICU, radiology: high-resolution 3D color fMRI brain scan = 4.5PB • Telepresence: master classes, virtual classrooms, tele-psychiatry• High performance video delivery: Uncompressed HD, Cinegrid• Disaster Recovery and distributed storage

40

Tier 1 to Tier 2 Traffic

The New Requirements

• High performance applications are dependent on high performance networks

• Networks must be fast, reliable, scalable, have flexible architectures, be cost effective, capable of delivering multiple services across multiple network layers, easy to operate and maintain, and have a view towards the future

• Enable the user – the network as a service

42

Questions?

43

44