Advanced Open Communication Exchange Facilities:

Operations, Economics, Overview of Selected

Exchanges

Joe Mambretti, Director, (j-mambretti@northwestern.edu)

International Center for Advanced Internet Research (www.icair.org)

Northwestern University

Director, Metropolitan Research and Education Network (www.mren.org)

Co-Director, StarLight, PI-iGENI, PI-OMNINet (www.startap.net/starlight)

GLORIAD in Asia: A Workshop To Design and Develop Advanced Science

Research Communications Exchange Facilities In Asia

University of Hong Kong

January 19-20, 2011

Sloan Digital Sky

Survey

www.sdss.org

Globus Alliance

www.globus.org

LIGO

www.ligo.org TeraGrid

www.teragrid.org

ALMA: Atacama

Large Millimeter

Array

www.alma.nrao.edu

CAMERA

metagenomics

camera.calit2.net

Comprehensive

Large-Array

Stewardship System

www.class.noaa.gov

DØ (DZero)

www-d0.fnal.gov

ISS: International

Space Station

www.nasa.gov/statio

n

IVOA:

International

Virtual

Observatory

www.ivoa.net

BIRN: Biomedical

Informatics Research

Network

www.nbirn.net

GEON: Geosciences

Network

www.geongrid.org

ANDRILL:

Antarctic

Geological

Drilling

www.andrill.org

GLEON: Global Lake

Ecological

Observatory

Network

www.gleon.org Pacific Rim

Applications and

Grid Middleware

Assembly

www.pragma-

grid.net

CineGrid

www.cinegrid.org Carbon Tracker

www.esrl.noaa.gov/

gmd/ccgg/carbontrack

er

XSEDE

www.xsede.org

LHCONE

www.lhcone.net

WLCG

lcg.web.cern.ch/LCG/publi

c/

OOI-CI

ci.oceanobservatories.org

OSG

www.opensciencegrid.org

SKA

www.skatelescope.o

rg

NG Digital

Sky Survey

ATLAS

Compilation By Maxine Brown

StarLight International/National Communications

Exchange Facility

Abbott Hall, Northwestern University‟s Chicago Campus

StarLight – “By Researchers For Researchers”

Abbott Hall, Northwestern University‟s

Chicago Campus View from StarLight

StarLight Is An Advanced Communication Exchange Facility And Prototyping Platform Built On An Optical infrastructure Optimized for High-Performance Applications World‟s Highest Capacity Exchange First Facility Enabling Interoperability At L1, L2, L3 Currently Implementing A Multi-100 Gbps Exchange

StarLight Infrastructure

StarLight is a large research-friendly co-location facility with

space, power and fiber that is being made available to

university and national/international network collaborators

as a point of presence in Chicago

The StarLight Facility Supports Many Different

Private Networks • International R&E Networks

• National R&E Production Networks

• National R&E Testbed Networks

• Other R&E National / Regional/ Local Production

Networks

• State R&E Networks

• Commercial Networks

• Prototypes

• Experimental Networks

•

• An Advanced Network for Advanced Applications

• Designed in 1993; Initial Production in 1994, Managed at L2 & L3

• Created by Consortium of Research Organizations -- over 20

• Partner to STAR TAP/StarLight, I-WIRE, NGI and R&E Net Initiatives, Grid and Globus Initiatives etc.

• Model for Next Generation Internets

• Developed World‟s First GigaPOP

• Next – the “Optical MREN”

• Soon - Optical „TeraPOP‟ Services

STAR-TAP 1997-2001

• World‟s First Communications Exchange for

International Research and Education Networks

• Original Design Based On Original MREN

Infrastructure

• ATM PVCs

• One L2 Hop Among Sites

iCAIR: Founding Partner of the Global Lambda Integrated Facility

Available Advanced Network Resources

Visualization courtesy of Bob Patterson, NCSA; data compilation by Maxine Brown, UIC.

www.glif.is

GLIF is a consortium of institutions, organizations, consortia and country

National Research & Education Networks who voluntarily share optical

networking resources and expertise to develop the Global LambdaGrid for the

advancement of scientific collaboration and discovery.

GLIF 2011

GLIF 2011

GLIF 2011

IRNC TL/SL Deliverables

• Continue Enabling Multi-National Application and

Middleware Experiments Through Innovative Services and

Technologies On International Networks: – High-Performance Digital Media Network (HPDMnet)

– iGENI: the GENI-funded international GENI project* ##

– SAGE: connecting people and their data at high-res*

– CineGrid: it‟s all about visual communications

– GreenLight International: less watts/terabyte*

– Science Cloud Communication Services Network (SCCSnet)*: the

impending disruption

• Build Cooperative National and International Partnerships*

• Provide New Services, Including Many with Industrial

Partners

• Capitalize On Other Emerging Opportunities*

## Now, In Part, A CISE/OCI Partnership!!

*Currently also funded by various NSF awards to UCSD/UIC/NU

TransLight/StarLight

Collaborates with All IRNC/GLIF Initiatives

With US HEP/LHC researchers, do trials

to move multi-gigabit traffic between

CERN and Brazil (Geneva to Amsterdam;

via LHCnet to Chicago; via CAVEwave to

DC; via AtlanticWave to Miami;

via AmLight-East to Brazil

Connect to TransLight/PacificWave

in Seattle via TransLight (Cisco

Research Wave deployed on NLR)

Provide GLORIAD via StarLight

with services to support multi-

gigabit US traffic to partners in

Russia, Netherlands, Nordic

countries, Asia

ACE

CA*net4 has 10x10Gb and Equipment at

StarLight

Source: CANARIE

SURFnet6 National Optical R&E Network

• High Performance Optical Switching

• Numerous 10 Gbit/s Lightpaths

• Dynamic Provisioning

• 500,000 Users

• 84 Institutes

TransLight/Pacific Wave 10GE Wave Facilitates US West Coast Connectivity

Developing a distributed exchange facility on the US West Coast (currently

Seattle, Sunnyvale and Los Angeles) to interconnect international and US

research and education networks

www.pacificwave.net/participants/irnc/

AARNet 2011

JGN-X Network Topology

National Institute for Information

Communication Technology (NICT)

National International Testbed

10 Gbps Connection via

GLIF/TransLight StarLight

KREONet2, KRLight

Europe

Kwangju Busan

Suwon Chonan

Jeonju Chang-

won

Pohang

K R E O N E T

APII

Seoul

KREONet2

High-Performance S&T Facilities

(High-Performance Cluster/Supercomputers, Storage,

Experimental Facilities, Visualization, Access Grid, DB Servers, etc.)

e-Science based KREONET/KREONet2/SuperSIReN

→ joining global Cyber Infrastructure

GEANT

Singapore/SingAREN

Daegu

China/CSTNET

Incheon

Japan/SINET

APII

TEIN

Daejeon SuperSIReN

Source: Kreonet

SuperJANET

R&D Network

GEANT

UKLight

London

StarLight

Chicago

NetherLight

Amsterdam

CERN

CzechLight Local Research

Equipment

International

Point-of-Access

CA*net

NRNs

UKLight via IRNC at StarLight

10Gb/s

10Gb/s

10Gb/s

10Gb/s

10Gb/s

10 Gb/s

Source: Peter Clarke, David Salmon, UKLight

CESNET

America‟s Lightpaths (AmLight)

GLIF SA 2011

TeraGrid=> XSEDE

JGI SNLL

LBNL

SLAC Fix-W

YUCCA MT

PNNL

LIGO

LANL

SNLA

Allied Signal

ARM

Allied Signal

NOAA

OSTI ORAU

SRS

ORNL JLAB

MAE-E

PPPL

ANL-DC

INEEL-DC

ORAU-DC LLNL/LANL-DC

MIT

ANL

BNL

FNAL

AMES

Nevis Yale

4xLAB-DC NERSC PAIX-E Mae-W

ALB HUB

DC HUB

CHI HUB SNV HUB

LLNL

GA

SDSC

Japan

GTN&NNSA

SAN

International (high speed)

OC192 (10G/s optical)

OC48 (2.5 Gb/s optical)

Gigabit Ethernet (1 Gb/s)

OC12 ATM (622 Mb/s)

OC12

OC3 (155 Mb/s)

T3 (45 Mb/s)

T1-T3

T1 (1 Mb/s)

Office Of Science Sponsored (22)

NNSA Sponsored (12)

Joint Sponsored (3)

Other Sponsored (2 LIGO, NOAA)

Laboratory Sponsored (6)

QWEST

ATM

42 end user sites

ESnet IP

GEANT

- Germany

- France

- Italy

- UK

- etc

CA*net4

CERN

MREN

Netherlands

Russia

StarTap

Taiwan (ASCC)

CA*net4

KDDI (Japan)

France

Switzerland

Taiwan (TANet2)

Australia

CA*net4

Taiwan (TANet2)

Singaren

DOE‟s ESnet Connects Facilities and Collaborators at StarLight

Sinet (Japan)

Japan –

Russia(BINP)

Source: DOE

DOE‟s UltraScience Net is at StarLight

Europe (GÉANT)

Asia-Pacific

New York

Washington

DC

CERN (30 Gbps, each

path)

Albuquerque

Au

str

ali

a

San Diego

LA

Denver

South America (AMPATH)

South America (AMPATH)

Canada (CANARIE)

Canada (CANARIE)

Asia Pacific GLORIAD (Russia and

China)

Boise

Jacksonville

Tulsa

Boston

Science Data

Network Core

IP Core

Au

str

ali

a

Core network fiber path is

~ 14,000 miles / 24,000 km

16

25

mile

s / 2

545

km

2700 miles / 4300 km

Production IP core (10Gbps)

SDN core (20-30-40Gbps)

MANs (20-60 Gbps) or

backbone loops for site access

International connections

IP core hubs

Primary DOE Labs

SDN (switch) hubs

High speed cross-connects with Ineternet2/Abilene

Possible hubs

El Paso

ESnet4 Planned Configuration 160-400 Gbps in 2011-2012

DOE ESnet Advanced Networking Initiative: 100 Gbps

DOE Office of Science ESnet ANI Testbed

StarLight

LHCnet 2008

NIH Biomedical Informatics Research Network (BIRN)

International Federated Repositories

www.nbirn.net

BIRN Collaboratory today: Enabling collaborative research at 28 research institutions comprised of 37 research groups.

NISN

NGIX West

New Generation IP Connections to

West Coast Regional networks,

Tier 1 ISP networks,

the PacRim, Japan, and Australia

MAE West

Most Important IP Connections to

West Coast Regional networks,

Tier 1 ISP networks,

the PacRim, Japan, and Australia

Commercial Ameritech NAP

Starlight

Most Important Connections to

Mid-West & SouthEast

Regional networks, Tier 1 ISP networks

NGIX East

New Generation IP Connections to

East Coast Regional networks,

Tier 1 ISP networks, and Europe

MAE East

Most Important Connections to

East Coast Regional networks,

Tier 1 ISP networks, and Europe

NASA‟s NISN is at StarLight

Source: NASA

DREN Network Is At StarLight

UltraLight Network: PHASE III

• Move into production

• Optical switching fully enabled amongst primary sites

• Integrated international infrastructure

Source:

UltraLight Network

Distributed US Atlas Computing Network

Source: John Silvester

Starlight

MREN

Village of Hoffman

Estates

NIU Hoffman

Estates

City of Naperville

NIU Naperville

City of Braceville

Repeater Station

City of Pontiac

OSF St. James

JW Albrecht MC

City of Chenoa

OSF Medical Group

City of Eureka

Eureka Community

HospitalTown and Country

Rural Health Care

ClinicCity of Peoria

Pekin Hospital

Lincoln

Abraham Lincoln

Memorial Hospital

Lincoln

Rural Health Clinic

Springfield

St. Johns Hospital

Springfield

Memorial Medical

Center

City of Litchfield

St. Francis Hospital

Litchfield Family

Practice Center

City of Staunton

Community

Memorial HospitalStaunton Family

Practice

City of East St. Louis

Kenneth Hall

Regional HospitalGermantown

Clinton Co. Rural

Health Clinic

City of Centralia

St. Mary’s Hospital

City of Salem

Salem Township

Hospital

City of Effingham

St. Anthony’s Memorial

Hospital

Mid-Ill Medical Care

Assoc. LLC

City of Mattoon

Sarah Bush Lincoln

Health CenterCity of Paris

Paris Community

Hospital

Paris Family

Medical Center

City of Tuscola

Carle Clinic

City of Urbana

Provena Covenant

Medical Center

City of Paxton

The Paxton Clinic

City of Onarga

The Onarga Clinic

City of Kanakee

Provena St. Mary’s

Hospital

Riverside

Medical Clinic

City of Normal

ISU

BroMenn

Health Care

OSF St. Joseph

Medical Center

City of Decatur

Decatur Memorial

Hospital

St. Mary’s Hospital

City of Jacksonville

Passavant Area

Hospital

Pittsfield

Illini Community

Hospital

City of Quincy

Blessing Hospital

City of Warsaw

Hamiton-Warsaw

Clinic

City of Carthage

Memorial Hospital

Women and Family

Medical Care

City of Macomb

Sarah Bush Lincoln

Health Center

City of Canton

Graham Hospital

Coleman Rural

Health Clinic

Rockford

NIU Rockford

City of DeKalb

NIU DeKalb

TriRivers

Health Network

Freeport

FHN Memorial

Hospital

Galena

Stauss Hospital

Rochelle

Rochelle

Community Hosptial

City of Rochelle

Your Family

Doctor

Dixon

Katherine Shaw

Bethea Hosptial

Sterling/Rock Falls

CGH Medical Center

Janet Waddles

Center

Belvidere

Northwest Suburban

Community Hosp

UIUC Medical

Center

Kishwaukee

Community

Hospital

Ben Gorden

Center

Illinois Rural HealthNet

Logical Network Map

U of I

City of Malta

Tri-County Nursing

Internet 2

City of Belleville

Memorial Hospital

City of Danville

Provena USMC

Danville

Pediatric Center

DeKalb County

Health DuPage National

Tech Park

(ISP Services)

Stockton

FHN Family

HealthCare

Lena

FHN Family

HealthCare

Warren

FHN Family

HealthCare

Orangeville

FHN Family

Healthcare

Pecatonica

FHN Family

Healthcare

Byron

Swedish American

Medical Group

Davis Junction

Swedish American

Medical

Oregon

FHN Family

Healthcare

Forreston

FHN Family

Healthcare

Lannark

FHN Family

Healthcare

Mt. Carroll

FHN Family

Healthcare

Savanna

FHN Family

Healthcare

Harvard

Mercy-Harvard

Hospital

Sandwich

Valley West

Community Hospital

Morrison

Morrison Community

Hosptial

Amboy

Sinnissippi

Center

Mendota

Mendota

Community Hospital

Princeton

Perry Memorial

HospitalKewanee

Kewanee Hospital

Genesso

Hammond-Henry

Hospital

Galva

Regional Family

Health Center

Galesburg

St. Mary’s Medical

Center

Mercer

Mercer County

Hospital

Monmouth

Community Medical

Center

Table Grove

Family Practice

Astoria

Community Medical

Center

Rushville

Sarah D. Culbertson

Memorial Hosptial

Mason

Mason District

Hospital

Winchester

Winchester Family

Practice

Hopedale

Hopedale Medical

Complex

Clinton

Dr. John Warner

Hospital

Pana

Pana Community

Hospital

Talyorville

St. Vincent

Memorial Hosptial

Carlinville

Carlinville Area

Hosptial

Greenfield

Boyd Fillager Clinic

Carrollton

Thomas Boyd

Memorial Hospital

Hillsboro

Hillsboro Area

Hospital

Vandalia

Fayette County

Hosptial

Highland

St. Joseph’s Hospital

Newton

Brush Creek Medical

Center

Robinson

Crawford

Memorial

Hospital

City of Urbana

Carle Foundation

Hosptial

Monticello

John and Mary E.

Kirby Hosptial

Gibson City

Gibson City Area

Hospital

Cissna Park

Cissna Park Medical

Clinic

Hoopeston

Hoopeston

Community Memorial

Hospital

Flora

Clay County

Hosptial

Fairfield

Fairfield Memorial

Hospital

Olney

Richland Memorial

Hospital

Lawrenceville

County Memorial

Hospital

Mount Carmel

Wabash General

Hospital District

Mount Vernon

Crossroad

Community Hosptial

McLeansboro

Hamilton Memorial

Hosptial

Eldorado

Ferrell Hosptial

Benton

Franklin Hospital

Nashville

Washington County

Hospital

Pinckneyville

Community

HospitalSparta

Community Hospital

Red Bud

Regional Hosptial

Chester

Memorial Hospital

DuQuoin

Marshall Browning

Hospital

Murphysboro

St. Joseph Memorial

Hospital

StarWave: A Multi-100 Gbps Facility

• StarWave, A New Advanced Multi-100 Gbps Facility and Services

Will Be Implemented Within the StarLight

International/NationalCommunications Exchange Facility

• StarWave Is Being Funded To Provide Services To Support Large

Scale Data Intensive Science Research Initiatives

• Facilities Components Will Include:

• An ITU G. 709 v3 Standards Based Optical Switch for WAN

Services, Supporting Multiple 100 G Connections

• An IEEE 802.3ba Standards Based Client Side Switch,

Supporting Multiple 100 G Connections, Multiple 10 G

Connections

• Multiple Other Components (e.g., Optical Fiber Interfaces,

Measurement Servers, Test Servers

iGENI: The International GENI

• The iGENI Initiative Will Design, Develop, Implement, and Operate a Major New National and International Distributed Infrastructure.

• iGENI Will Place the “G” in GENI Making GENI Truly Global.

• iGENI Will Be a Unique Distributed Infrastructure Supporting

Research and Development for Next-Generation Network

Communication Services and Technologies.

• This Infrastructure Will Be Integrated With Current and Planned

GENI Resources, and Operated for Use by GENI Researchers

Conducting Experiments that Involve Multiple Aggregates At

Multiple Sites.

• iGENI Infrastructure Will Connect Its Resources With Current

GENI National Backbone Transport Resources, With Current

and Planned GENI Regional Transport Resources, and With

International Research Networks and Projects,

•

A Few Items On Economics

• The StarLight Design Was Undertaken To Ensure That

All Aspects Could Be Addressed At The Lowest

Possible Cost

• Non-Carrier Environment

• Open Access

• No Charges For Riser Fiber

• No Charges For Cross Connections

• No Miscellaneous „Port” Charges

• Low Cost NOC Operations

• Etc

www.startap.net/starlight

Thanks to the NSF, DOE, DARPA

Universities, National Labs,

International Partners,

and Other Supporters