ICFA Standing Committee onICFA Standing Committee onInterregional Connectivity (SCIC)Interregional Connectivity (SCIC)

Harvey B. Newman Harvey B. Newman California Institute of TechnologyCalifornia Institute of Technology

CHEP2009, Prague, March 23 2009CHEP2009, Prague, March 23 2009

Global Networks for HEP in 2009

World Internet Connection Density 2008http://www.chrisharrison.net/projects/InternetMap/

The World At Nighthttp://www.bertc.com/subfour/truth/nightworld.htm

http://monalisa.caltech.edu:8080/Slides/Public/SCICReports2009Finalhttp://monalisa.caltech.edu:8080/Slides/Public/SCICReports2009Final

SCIC in 2009 http://cern.ch/icfa-scic

SCIC in 2009 http://cern.ch/icfa-scic

Three 2009 Reports: An Eventful YearThree 2009 Reports: An Eventful YearRapid Progress, Dawn of an Era; Deepening Digital DivideRapid Progress, Dawn of an Era; Deepening Digital Divide

Main ReportMain Report: “Networking for HEP”: “Networking for HEP” [HN, A. Mughal et al.][HN, A. Mughal et al.] Includes Updates on the Digital Divide, World Network Includes Updates on the Digital Divide, World Network

Status; Brief Updates on Monitoring, Advanced Status; Brief Updates on Monitoring, Advanced Technologies; Technologies; Focus on Digital Divide IssuesFocus on Digital Divide Issues

3939 Annexes: A World Network Overview Annexes: A World Network Overview Status and Plans of Nat’l & Regional Networks, HEP Labs, Status and Plans of Nat’l & Regional Networks, HEP Labs, & Optical Net Initiatives & Optical Net Initiatives [[35 Updated in 200935 Updated in 2009]]

Monitoring Working Group ReportMonitoring Working Group Report [R. Cottrell, U. Kalim] [R. Cottrell, U. Kalim] Also See: Also See: TERENA TERENA 2008 Compendium2008 Compendium ((www.terena.nlwww.terena.nl)): : R&E Networks in EuropeR&E Networks in Europe http://internetworldstats.comhttp://internetworldstats.com: Worldwide Internet Use: Worldwide Internet Use OECD Broadband PortalOECD Broadband Portal http://www.oecd.org/sti/ict/broadbandhttp://www.oecd.org/sti/ict/broadband

Broadband: the new Digital Divide ?Broadband: the new Digital Divide ? SCIC 2003 Digital Divide ReportSCIC 2003 Digital Divide Report [A. Santoro et al.][A. Santoro et al.]

North Am.

Revolutions in Networking Revolutions in Networking

Explosion of BW Use: Explosion of BW Use: Now ~4,000 PB/moNow ~4,000 PB/mo

Raw capacity still Raw capacity still mostly unusedmostly unused

Rise of broadbandRise of broadband Rise of Video + Mobile Rise of Video + Mobile

Traffic: ~20 ExabytesTraffic: ~20 ExabytesPer mo. (64%) by 2013Per mo. (64%) by 2013

Web 2.0: Billions Web 2.0: Billions of Web Pages, of Web Pages, embedded apps.embedded apps. Facebook, Twitter, Facebook, Twitter,

iPhone/Gphone, iPhone/Gphone, GEarth; SkypeGEarth; Skype

Beginnings of Web 3.0: Beginnings of Web 3.0: streaming content; streaming content; ubiquitous informationubiquitous information

11stst Revolution: “Long Dawn” of the Information Age Revolution: “Long Dawn” of the Information Age

1.6B1.6B Internet Users; 400+M with Broadband (12/31/08) Internet Users; 400+M with Broadband (12/31/08)http://internetworldstats.com

%0 10 20 30 40 50 60 70 80

73%

49%24%

17%22%

5.4%

60%

Asia

Latin Am.

Africa

Mid. East

Australasia/Oceana

World Av. 23%

Broadband: 80M Each in the US and ChinaBroadband: 80M Each in the US and China

World Penetration Rates (12/31/08)

Europe

SCIC Report 2009 - Main Trends Accelerate:SCIC Report 2009 - Main Trends Accelerate: Dark Fiber Nets, Dynamic Circuits, 40-100GDark Fiber Nets, Dynamic Circuits, 40-100G

SCIC Report 2009 - Main Trends Accelerate:SCIC Report 2009 - Main Trends Accelerate: Dark Fiber Nets, Dynamic Circuits, 40-100GDark Fiber Nets, Dynamic Circuits, 40-100G

Current generation of 10 Gbps network backbones and major Int’l Current generation of 10 Gbps network backbones and major Int’l links arrived in 2002-8 in US, Europe, Japan, Korea; Now links arrived in 2002-8 in US, Europe, Japan, Korea; Now China China

Bandwidth Growth: from 16X to Bandwidth Growth: from 16X to 10,000X in 7 Yrs. >> Moore’s Law10,000X in 7 Yrs. >> Moore’s Law Proliferation of 10G links across the Atlantic & Pacific since 2005Proliferation of 10G links across the Atlantic & Pacific since 2005

Bandwidth for LHC well above 100 Gbps, in aggregateBandwidth for LHC well above 100 Gbps, in aggregate Rapid Spread of “Dark Fiber” and DWDM: Emergence of Rapid Spread of “Dark Fiber” and DWDM: Emergence of Continental, Continental,

Nat’l, State & MetroNat’l, State & Metro N X 10G “Hybrid” Networks in Many NationsN X 10G “Hybrid” Networks in Many Nations Demand may still exceed capacity by ~2010, in some regionsDemand may still exceed capacity by ~2010, in some regions

Point-to-point “Light-paths” for HEP and “Data Intensive Science”Point-to-point “Light-paths” for HEP and “Data Intensive Science” Now Dynamic Circuits, and Managed Bandwidth ChannelsNow Dynamic Circuits, and Managed Bandwidth Channels

Technology continues to drive Performance Higher, Costs Lower Technology continues to drive Performance Higher, Costs Lower Commoditization of GE now 10 GE ports on serversCommoditization of GE now 10 GE ports on servers Multicore processors with Multi-Gbyte/sec interconnectsMulticore processors with Multi-Gbyte/sec interconnects Cheaper and faster storage ($ < 100/Tbyte); 100+ Mbyte/sec disksCheaper and faster storage ($ < 100/Tbyte); 100+ Mbyte/sec disks

TransitionTransition to 40G, 100G links to 40G, 100G links from 2011 (on land), ~2012 (sub-sea) from 2011 (on land), ~2012 (sub-sea) OutlookOutlook:: Continued growth in bandwidth deployment & useContinued growth in bandwidth deployment & use

Core NREN Capacity of EU/EFTA

Countries 2004-2008 TERENA Compendium 2008:

www.terena.org/activities/compendium/100G

10G

1G

0.1G

20G

atbe

bgcy

czdk

ee

frde

grhu

isie

it ltlu

mtnl

noplfi lv

sksi

esse

chuk

ptro

LHC Optical Private Network (The LHCOPN)

LHC Optical Private Network (The LHCOPN)

Edoardo Martelli (CERN)

Close collaboration with CERN & GEANT2 (11 10G Links)

Working with GEANT2 to achieve fiber diversity for European circuits to/from CERN

Inter-Tier1 links for resilience, redundancy:

BNL, FNAL: US LHCNetCNAF, SARA, GridKa, IN2P3:

Cross-border dark fiberRAL (UK): 2nd 10G in ProcessNDGF, ASGC, Taiwan, TRIUMF:

Reduced BW backupPIC (Spain): Still no backup

Evolution to N X 10 Gbps: Fermilab, BNL, US LHCNet; SURFnet, GARR-X

GÉANT2 Pan-European BackboneDark Fiber Core

Among 19 Countries: Austria Belgium Croatia Czech Republic Denmark Finland France Germany Hungary Ireland Italy Netherlands Norway Slovakia Slovenia Spain Sweden Switzerland United KingdomC. Stover (DANTE)

34 NRENs, ~30M Users; 50k km Leased Lines 12k km Dark Fiber; Point to Point ServicesGN3 Next Gen. Network Proposed 9/2008

Projected Start Q2 2009

SNLL

PNNL

LIGO

INEEL

LANL

MIT

ANLAMES

LLNL

GA

DOE-ALB

JGI

LBNL

SLACCHI

IARC

DenverLNOC

NERSC

BNL

FNAL

CH

I-SL

NYC

NYC

CERN

Amsterdam

By 2006

US LHCNet 10Gb/sUS LHCNet 20 Gb/sInternational (high speed)10 Gb/s SDN core10G/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less

ESnet4 IP Core (10 Gbps)

ESNet4 Science Data Net Core (n*10 Gbps)

Redundant “light-paths” to BNL and FNAL; dark fiber to FNAL

Connections to ESnet MANs in NYC & Chicago 10 Gbps peerings with Abilene (2) and GEANT2

US LHCNet (4x10Gbps)

US LHCNet + ESnet4 Today

SURFNet

In 2007

US-LHCNet Plan 2008-10: 40, 60, 80

GbpsNY-CHI-GVA-AMS

In 2008

A Global Partnership of R&E Networks and Advanced R&D Projects Supporting the LHC Program

14 to 16 10G Trans- Atlantic Links in 2009

Partnership with Internet2, NLR, ESnet in the US

Computing Model Progress CMS Internal Review of Software and Computing

CMS Data Transfer Volume (May – Aug. 2007)

10 PetaBytes Transferred Over 4 Mos. = 8.0 Gbps Avg. (15 Gbps Peak)

2008: 9-12 Gbps T1-T2 Flows

To UCSD, Nebraska, Caltech

15Log Plot of ESnet Monthly Accepted Traffic, January 1990 – December 2008

Remarkable Historical ESnet Traffic Trend Cont’d in 2008T

erab

ytes

/ m

on

th Oct 19931 TBy/mo.

Aug 1990100 MBy/mo.

Jul 199810 TBy/mo.

38 months

57 months

40 months

Nov 2001100 TBy/mo.

Apr 20061 PBy/mo.

53 months

July 201010 PBy/mo.

J. Metzger

ESnet Traffic Increases by10X Every 47 Months on Avg.

10 PBytes/mo. By ~July 2010Equal to 30 Gbps Continuous

Bandwidth Roadmap for Major Links (in Gbps): US LHCNet Example

Bandwidth Roadmap for Major Links (in Gbps): US LHCNet Example

Year Production Experimental Remarks

2001 0.155 0.622-2.5 SONET/SDH

2002 0.622 2.5 SONET/SDH DWDM; GigE

Integ.

2003 2.5 10-20 DWDM; 1 + 10 GigE

Integration

2005-6 10-20 2-10 X 10 Switch; Provisioning

2007-8 3-4 X 10 ~10 X 10; 100 Gbps

1st Gen. Grids

2009-10 6-8 X 10

~20X10, 5X40 or ~2 X 100

100 Gbps Switching

2011-13 ~12 X 10 to 0.25 Tbps

~10 X 100 2nd Gen Grids Terabit Networks

2014-6 ~Terabit ~MultiTbps ~Fill One Fiber

Paralleled by ESnet Roadmap for Data Intensive Sciences

We are progressing along this roadmap

Science Network Requirements Aggregation SummaryScience Drivers

Science Areas / Facilities

End2End Reliability

Near Term End2End

Band width

5 years End2End

Band width

Traffic Characteristics

Network Services

HEP:

LHC (CMS and Atlas)

99.95+%

(Less than 4 hours

per year)

73Gbps 225-265 Gbps

•Bulk data•Coupled analysis workflows

• Collaboration services

• Grid / PKI• Guaranteed

bandwidth• Monitoring / test

tools

NP:

CMS Heavy Ion

- 10Gbps (2009)

20 Gbps •Bulk data • Collaboration services

• Deadline scheduling• Grid / PKI

NP:

CEBF (JLAB)

- 10Gbps 10 Gbps •Bulk data • Collaboration services

• Grid / PKI

NP:

RHIC

Limited outage

duration to avoid analysis pipeline

stalls

6Gbps 20 Gbps •Bulk data • Collaboration services

• Grid / PKI• Guaranteed

bandwidth• Monitoring / test

tools

Immediate Requirements and Drivers for ESnet4

HENP: ~300 Gbps by 2013

ESnet Science Network Requirements Aggregation SummaryScience DriversScience Areas / Facilities

End2End Reliability

Near Term End2End

Band width

5 years End2End

Band width

Traffic Characteristics Network Services

BES:Chemistry and Combustion

- 5-10Gbps 30 Gbps • Bulk data• Real time data

streaming

• Data movement middleware

BES:Light Sources

- 15Gbps 40-60 Gbps

• Bulk data• Coupled simulation

and experiment

• Collaboration services• Data transfer facilities• Grid / PKI• Guaranteed BW

BES:Nanoscience Centers

- 3-5Gbps 30 Gbps •Bulk data•Real time data streaming•Remote control

• Collaboration services• Grid / PKI

Fusion ES:Int’l Collaboration

- 100Mbps 1 Gbps • Bulk data • Enhanced collaboration services

• Grid / PKI• Monitoring / test tools

Fusion ES:Instruments and Facilities

- 3Gbps 20 Gbps • Bulk data• Coupled simulation

and experiment• Remote control

• Enhanced collaboration service

• Grid / PKI

Fusion ES:Simulation

- 10Gbps 88 Gbps • Bulk data• Coupled simulation

and experiment• Remote control

• Easy movement of large checkpoint files

• Guaranteed bandwidth• Reliable data transfer

Fusion + BES + Bioinformations ~Equal to HEP

Cle

vela

nd

Primary DOE Labs

IP core hubsSDN hubs

Europe(GEANT)

Asia-Pacific

New YorkChicago

Washington DC

Atl

anta

Seattle

Albuquerque

Au

stra

lia

San Diego

LA

Production IP core (10Gbps)

SDN core (20-30-40Gbps)

MANs (20-60 Gbps) or backbone loops for site access

International connections

SunnyvaleDenver

South America(AMPATH)

South America(AMPATH)

Canada(CANARIE)

US LHCNet to CERN (60-80; 120-180 Gbps)

Canada(CANARIE)

Europe(GEANT)

Asi

a-Pac

ific

Asia Pacific

High Speed Cross connects with Ineternet2/Abilene

GLORIAD (Russia and

China)

Boise

Houston

Jacksonville

Possible hubs

Tulsa

Boston

Science Data Network Core

IP Core

Kansa

s City

Au

stra

lia

Fiber path is ~ 14,000 miles / 24,000 km

www.es.net/ESNET4ESnet4 50-60 Gbps by 2009-10; 500-600 Gbps 2011-12

100 Gbps waves “production-ready” within ~1.5-2 years (Infinera; Also CIENA)

Computing Model Progress CMS Internal Review of Software and Computing

Implementation: US LHCNet Plan Phase 8 Implementation: US LHCNet Plan Phase 8 (2014): Transition to Full Use of 100G(2014): Transition to Full Use of 100G

Using OTU-4 Using OTU-4 (100G) Links(100G) Links+ Next-Gen. + Next-Gen.

Optical MuxesOptical Muxes

4 X 100G4 X 100GTrans-Trans-

AtlanticAtlantic

FollowingFollowingan 8 Phasean 8 Phase

Plan Plan 2007-20142007-2014

US LHCNet Bandwidth Roadmap Matches LHC Storage Roadmap

US LHCNet Bandwidth Roadmap Matches LHC Storage Roadmap

20082009

20102011

20122013

Atlas Disk Ratio wrt 2008

AVG Disk Ratio wrt 2008

USLHCNet Bandwidth

CMS Disk Ratio wrt 2008

0

1

2

3

4

5

6

7

8

9

10Atlas Disk Ratio wrt 2008

AVG Disk Ratio wrt 2008

USLHCNet Bandwidth

CMS Disk Ratio wrt 2008

2008 2009 2010 2011 2012 2013

Atlas Disk Ratio wrt 2008 1 1.85 3.39 4.77 6.16 9.51

AVG Disk Ratio wrt 2008 1 1.55 2.63 3.62 4.62 6.60

USLHCNet BW Ratio [40G] 1 [60] 1.50 [80] 2.0 [120] 3.0 [180] 4.5 [260] 6.5

CMS Disk Ratio wrt 2008 1 1.25 1.87 2.47 3.08 3.69

40G in 200840G in 2008to to

260G in 2013:260G in 2013:

6.5X in 5 Yrs.6.5X in 5 Yrs.

SlowerSlower than than historical historical

trends:trends:[20-100X][20-100X]

Computing Model Progress CMS Internal Review of Software and Computing

Downward Price Evolution

on TA 10G Links Continues

(-31%/Yr 2005-8)

Important for the LHC

There Are Price “Bumps” Along

the Way

Prices in Other Regions are

Higher

Question

How will 40 + 100G Link Prices

Evolve in

2010-14 ?

24

Use of Dark Fiber in NREN Backbones 2005 – 2008Greater or Complete Reliance on Dark Fiber

TERENA Compendium 2008: www.terena.org/activities/compendium/

2005 2008

SURFNet and NetherLight: 8000 Km Dark FiberFlexible Photonic Infrastructure

SURFNet and NetherLight: 8000 Km Dark FiberFlexible Photonic Infrastructure

Cross Border Fibers to Germany: X-Win; On to NORDUnetCross Border Fibers to Germany: X-Win; On to NORDUnet

5 Photonic Subnets

λ Switching Services

to 10G

Fixed or Dynamic

Lightpaths: LCG, GN2, EXPRES DEISA

CineGrid

5 Photonic Subnets

λ Switching Services

to 10G

Fixed or Dynamic

Lightpaths: LCG, GN2, EXPRES DEISA

CineGrid

Erik-Jan Bos

LCG/EGEEPOLTIER2

Distributed Tier2 (Poznan, Warsaw, Cracow) Connects to Karlsruhe Tier1

Cross Border Dark Fiber Links

to Russia, Lithuania,

Belarus, Czech Republic,

and Slovakia

R. Lichwala2 X 10G Among 20 Major University Centers

Existing 1Q 20094Q 2009

POLAND: PIONIER 6000 km Dark Fiber Network in 2009

POLAND: PIONIER 6000 km Dark Fiber Network in 2009

Czech Republic: CESNET2 Reconfigurable Optical Backbone in 2009

Czech Republic: CESNET2 Reconfigurable Optical Backbone in 2009

2500+ km 2500+ km Dark FibersDark Fibers(since 1999)(since 1999)

N X 10 GbE N X 10 GbE Light-PathsLight-Paths

10 GbE CBDF10 GbE CBDFSlovakiaSlovakiaPolandPolandAustriaAustria

NetherlightNetherlightGEANT2GEANT2

Czech Tier2:Czech Tier2: 1 Gigabit Lightpaths to the Tier1s 1 Gigabit Lightpaths to the Tier1s at Fermilab, BNL, Karlsruhe and Taiwan;at Fermilab, BNL, Karlsruhe and Taiwan;

Similar scheme: Similar scheme: it, br, nl, de, ….it, br, nl, de, ….

H. Sverenyak

2002 - 2004: Dark Fiber Links to Austria, Czech Republic, Poland

2005-6: Complete 1 GbE links to all main sites 2006: 10 GbE Cross-Border Dark Fiber to Austria & Czech

Republic; 8 X 10G over 224 km with Nothing In-Line Deonstrated

2007-8: Transition Backbone to 10G Done; All CB Dark Fibers to 10G

~10,000x Increase Since

2002

WeisHorvath http://www.sanet.sk/en/index.shtm

SLOVAK Academic Network January 2009:

Now ~All 10 GbE Switched Ethernet

SANET to Schools 1GE to

500 Schools

In 54 Cities

By 2012

The Emergence of “Hybrid” Networks The Emergence of “Hybrid” Networks With Dynamic Circuits with BW GuaranteesWith Dynamic Circuits with BW Guarantees

W. Johnston, ESnet W. Johnston, ESnet On Circuit-Oriented Network ServicesOn Circuit-Oriented Network Services

Traffic Isolation; Security; Deadline Scheduling; Fairness; High UltilizationTraffic Isolation; Security; Deadline Scheduling; Fairness; High Ultilization

US LHCNet Configuration (2H 2009)Non-stop Operation; Circuit-oriented Services US LHCNet Configuration (2H 2009)

Non-stop Operation; Circuit-oriented Services

Robust fallback at layer 1 + next-generation hybrid optical network: Robust fallback at layer 1 + next-generation hybrid optical network: Dynamic Dynamic circuit-oriented network services with BW guarantees circuit-oriented network services with BW guarantees

Emerging Emerging StandardsStandards

VCAT, LCASVCAT, LCAS

CIENA Core CIENA Core DirectorsDirectors

Equipment Equipment and link and link

RedundancyRedundancy

Also Internet2 and SINET3

(Japan)

2007 Outbound Traffic13.3 PBytes

77% on Circuits: Peak 18 Gbps

6.7 Gbps Average

2007 Outbound Traffic13.3 PBytes

77% on Circuits: Peak 18 Gbps

6.7 Gbps Average

Large Scale Flows Mostly Handled by Dynamic

CircuitsUsing Software by

Fermilab and Caltech

Large Scale Flows Mostly Handled by Dynamic

CircuitsUsing Software by

Fermilab and Caltech

Traffic on CircuitsTraffic on Circuits

CMS data transfer between FNAL and UNL using Internet2's

DCN and LambdaStation Software (FNAL + Caltech) Cumulative transfer volume (top) and data rates (bottom)

CMS data transfer between FNAL and UNL using Internet2's DCN and LambdaStation Software (FNAL + Caltech)

Cumulative transfer volume (top) and data rates (bottom)

Entire 50 TByte Tier2 Buffer Filled

in ~1 Day

Entire 50 TByte Tier2 Buffer Filled

in ~1 Day

9 Gbps PeaksUsing Circuits

on ESNet (Green)I2 DCN (Red)

9 Gbps PeaksUsing Circuits

on ESNet (Green)I2 DCN (Red)

NetherLight 2009: 37 Lambdas, 201 GbpsNetherLight 2009: 37 Lambdas, 201 Gbps

Convergence of Many Partners on Common Circuit Concepts

Internet2, ESnet, GEANT2, US LHCNet; cz,nl, ru, es, tw, kr, hk, in, nordic; jp

Convergence of Many Partners on Common Circuit Concepts

Internet2, ESnet, GEANT2, US LHCNet; cz,nl, ru, es, tw, kr, hk, in, nordic; jp

~512 CPU Cores and100 10GE NICs

in 1 Rack of Servers64 10GE Switch Ports;

~100 TB Disk

Research Partners: FNAL, BNL, Florida, Michigan, Brazil, Korea;

ESnet, NLR, FLR, Internet2, ESNet, CWave, AWave, IRNC,

CANARIE, SURFNET

40

6050

30

1020

70

0

2010

30

40

In (

Gb

ps)

Tra

ffic

: O

ut

SC08

Using FDT and FDT/dCache Storage to Storage

Max. 114 Gbps; 110 Gbps Sustained; 71 Gbps Outbound

Caltech and CIENA: 191 Gbps Avg., 199.90 Gbps Max on An OTU4

(Standard 100G) Wave at SC2008

10 X 10G Waves at the Caltech HEP Booth

Used Fully, in Both Directions with Caltech’s

FDT (TCP-Based Java Application)

Previewing the US LHCNet Transition to

4 X 100G by ~20141.02 Petabytes Overnight

SC08

Plots Courtesy the PingER Project

Data from http://internetworldstats.com

Digital Divide: North Vs. South East Vs. West

Work on the Digital Dividefrom Several PerspectivesWork on the Digital Dividefrom Several Perspectives

Share Information: Share Information: Monitoring,Monitoring, Tracking Tracking BW ProgressBW Progress;; Dark Fiber Projects & Pricing Dark Fiber Projects & Pricing Track Planning (focus on LHC) and Leading Edge ProgressTrack Planning (focus on LHC) and Leading Edge Progress Model Cases: Poland, Slovakia, Czech Rep., Brazil, China …Model Cases: Poland, Slovakia, Czech Rep., Brazil, China … Encourage Access to Dark Fiber; Modern technology choicesEncourage Access to Dark Fiber; Modern technology choices

Raise Awareness: Locally, Regionally & GloballyRaise Awareness: Locally, Regionally & Globally Digital Divide Workshops Digital Divide Workshops [Rio, Gaegu, Cracow, Sinaia, Mexico City][Rio, Gaegu, Cracow, Sinaia, Mexico City] Diplomatic Events: WSIS, RSIS, Bilateral: US-South Asia, EU-Africa,…Diplomatic Events: WSIS, RSIS, Bilateral: US-South Asia, EU-Africa,…

Technical Help with Modernizing the Infrastructure: Technical Help with Modernizing the Infrastructure: Provide Tools for Effective Use: Provide Tools for Effective Use: Data Transport, Monitoring, Data Transport, Monitoring,

Remote Collaboration and e-LearningRemote Collaboration and e-Learning Design, Commissioning, Development Design, Commissioning, Development India “Knowledge Network”, Brazil’s Nat’l and Int’l NetworksIndia “Knowledge Network”, Brazil’s Nat’l and Int’l Networks

Encourage, and Work on Inter-Regional Projects Encourage, and Work on Inter-Regional Projects GLORIAD,GLORIAD, Russia-China-Korea-US-Europe Optical Ring Russia-China-Korea-US-Europe Optical Ring Latin America: Latin America: CHEPREO/WHRENCHEPREO/WHREN (US-Brazil); (US-Brazil); RedCLARARedCLARA Mediterranean: Mediterranean: EUMEDConnect; EUMEDConnect; Asia-Pacific: Asia-Pacific: TEIN3; TEIN3; Asia:Asia: ORIENT ORIENT

SCIC Monitoring WG PingER (Also IEPM-BW) SCIC Monitoring WG PingER (Also IEPM-BW)

Measurements from 1995 OnMeasurements from 1995 OnReports link reliability & qualityReports link reliability & quality

Countries monitoredCountries monitored Contain Contain 95.5%95.5% of world pop. of world pop. 99% of World’s Internet Users99% of World’s Internet Users

890890 remote nodes at remote nodes at 650650 sites in sites in 165 nations165 nations; ; 45 monitoring nodes;45 monitoring nodes; 95 Sites in 50 African countries95 Sites in 50 African countries

Strong Collaboration with ICTP Strong Collaboration with ICTP Trieste and NUST/SEECS (Pakistan)Trieste and NUST/SEECS (Pakistan)

Excellent, Vital WorkExcellent, Vital Work

Monitoring & Remote Nodes (1/09)

Countries: N. America (3), Latin America (21), Europe (33), Balkans (10), Africa (50), Middle East (13), Central Asia (9), South Asia (8), East Asia

(4), SE Asia (10), Russia (1), China (1) and Oceania (4)

R. Cottrell

Number of Hosts MonitoredBy Region: 1998 – End 2008 Number of Hosts MonitoredBy Region: 1998 – End 2008

R. Cottrell

AfricaAfrica

End

SCIC Monitoring WG - Throughput Improvements 1998-2008

Bandwidth of TCP < MSS/(RTT*Sqrt(Loss)) Matthis et al., Computer Communication Review 27(3), July 1997Matthis et al., Computer Communication Review 27(3), July 1997

30% annual improvement

~14X Per 10 yrs

Progress: but Digital Divide is Mostly Maintained

Behind Europe3-6 Yrs: Russia, Latin America, SE Asia 9-12 Yrs: South Asia, Central Asia16 Years: Africa

South & Central Asia, EspeciallyAfrica are Falling Farther Behind:

100X-1000X Worse by 2019 ?

R. Cottrell

http://www-iepm.slac.stanford.edu/

98

99

00

01

02

03

04

05

06

07

08

09

10k

100k

1M

10M

Th

rou

gh

pu

t (b

ps)

Three undersea cables cut: traffic greatly disturbed between Europe and Asia/Near

East zone France Telecom observed Dec. 19 that 3 major underwater cables were cut:

“Sea Me We 4” at 7:28am, “Sea Me We3” at 7:33am and FLAG at 8:06am. Causes of the cut, located in the Mediterranean between Sicily and Tunisia,

on sections linking Sicily to Egypt, remain unclear. Most traffic between Europe and Asia rerouted through the USA. Traffic from Europe to Near East & Asia interrupted to greater or lesser extent.First estimate of percentage of out-of-service capacity): Saudi Arabia: 55% out of service Djibouti: 71% out of service Egypt: 52% out of service UAE: 68% out of service India: 82% out of service Lebanon: 16% out of service Malaysia: 42% out of service Maldives: 100% out of service Pakistan: 51% out of service Qatar: 73% out of service Syria: 36% out of service Taiwan: 39% out of service Yemen: 38% out of service Zambia: 62% out of service

www.orange.com/en_EN/press/press_releases/att00006557/CP_cables_19dec08_EN.pdf

Computing Model Progress CMS Internal Review of Software and Computing

Cable Cuts on Dec. 19.

Partial Recovery by Dec. 23

PingER Measurementsof the Impact of the Cuts on Throughput

Vs. Time

1 3 5 7 9 11 1315 17 19

21 23 9 2727

Brazil: RNP2 Next-Generation “Ipê” Backbone

Brazil: RNP2 Next-Generation “Ipê” Backbone

M. Stanton

New vs. OldBandwidth 70 to 300X

2007-9 Buildout of dark fiber nets in 27 cities: Connecting 200 Univ. & Institutes at 1 Gbps

2009: Upgrade 2.5G Sao Paulo –Miami Link to 2 X 10G (RNP+ANSP+NSF)10G to Grid-UNESP

Tier1, Tier2s in Rio and Sao Paulo

Dark Fiber (622 Mbps) across the Amazon Porto Velho to Manaus

Extending the Ipê Fiber footprint to the East and South

SCIC: Networks for HEPMain Conclusions for 2009

SCIC: Networks for HEPMain Conclusions for 2009

The major R&E networks serving HEP have progressed rapidly over The major R&E networks serving HEP have progressed rapidly over the last few years; to N X10G in many casesthe last few years; to N X10G in many cases Worldwide transition to 40G & 100G links will occur in 2010 – 14Worldwide transition to 40G & 100G links will occur in 2010 – 14

Our BW usage has kept pace; may soon outstrip the planned capacityOur BW usage has kept pace; may soon outstrip the planned capacity Groups in HEP have developed state of the art methods to use these Groups in HEP have developed state of the art methods to use these

networks most effectivelynetworks most effectively Widespread deployment, end-to-end monitoring, training requiredWidespread deployment, end-to-end monitoring, training required

Adapting the LHC Computing Models to fully exploit networks Adapting the LHC Computing Models to fully exploit networks would have a profound positive impact on the LHC programwould have a profound positive impact on the LHC program

It is more urgent than ever that we act It is more urgent than ever that we act to Close the Digital Divideto Close the Digital Divide To make physicists in all regions full partners in the discoveries To make physicists in all regions full partners in the discoveries

We are learning to do this effectively, in some cases, in partnership We are learning to do this effectively, in some cases, in partnership with many agencies & HEP groups:with many agencies & HEP groups: Central Europe Central Europe andand Brazil; India Brazil; India and and PakistanPakistan

But we are leaving other countries and regions behind, for example:But we are leaving other countries and regions behind, for example: the Middle East, the Rest of Latin America; Africa the Middle East, the Rest of Latin America; Africa

A great deal of work remainsA great deal of work remains