Harvey B NewmanHarvey B Newman

FAST Meeting, CaltechFAST Meeting, CaltechJuly 1, 2002July 1, 2002

http://l3www.cern.ch/~newman/HENPGridsNets_FAST070202.ppthttp://l3www.cern.ch/~newman/HENPGridsNets_FAST070202.ppt

HENP HENP Grids and NetworksGrids and Networks Global Virtual Organizations Global Virtual Organizations

Computing Challenges: Computing Challenges: Petabyes, Petaflops, Global VOsPetabyes, Petaflops, Global VOs

Geographical dispersion:Geographical dispersion: of people and resources of people and resources Complexity:Complexity: the detector and the LHC environment the detector and the LHC environment Scale: Scale: Tens of Petabytes per year of dataTens of Petabytes per year of data

5000+ Physicists 250+ Institutes 60+ Countries

Major challenges associated with:Major challenges associated with:Communication and collaboration at a distanceCommunication and collaboration at a distance

Managing globally distributed computing & data resources Managing globally distributed computing & data resources Cooperative software development and physics analysisCooperative software development and physics analysis

New Forms of Distributed Systems: Data GridsNew Forms of Distributed Systems: Data Grids

Four LHC Experiments: The Four LHC Experiments: The Petabyte to Exabyte Petabyte to Exabyte

ChallengeChallengeATLAS, CMS, ALICE, LHCBATLAS, CMS, ALICE, LHCB

Higgs + New particles; Quark-Gluon Plasma; CP ViolationHiggs + New particles; Quark-Gluon Plasma; CP Violation

Data storedData stored ~40 Petabytes/Year and UP; ~40 Petabytes/Year and UP; CPU CPU 0.30 Petaflops and UP 0.30 Petaflops and UP

0.1 to 1 Exabyte (1 EB = 100.1 to 1 Exabyte (1 EB = 101818 Bytes) Bytes) (2007) (~2012 ?) for the LHC Experiments(2007) (~2012 ?) for the LHC Experiments

All charged tracks with pt > 2 GeV

Reconstructed tracks with pt > 25 GeV

(+30 minimum bias events)

109 events/sec, selectivity: 1 in 1013 (1 person in a thousand world populations)

LHC: Higgs Decay into 4 muons LHC: Higgs Decay into 4 muons (Tracker only); 1000X LEP Data Rate(Tracker only); 1000X LEP Data Rate

LHC Data Grid HierarchyLHC Data Grid Hierarchy

Tier 1

Tier2 Center

Online System

CERN 700k SI95 ~1 PB Disk; Tape Robot

FNAL: 200k SI95; 600 TBIN2P3 Center INFN Center RAL Center

InstituteInstituteInstituteInstitute ~0.25TIPS

Workstations

~100-400 MBytes/sec

2.5-10 Gbps

0.1–10 GbpsPhysicists work on analysis “channels”

Each institute has ~10 physicists working on one or more channels

Physics data cache

~PByte/sec

~2.5-10 Gbps

Tier2 CenterTier2 CenterTier2 Center

~2.5-10 Gbps

Tier 0 +1

Tier 3

Tier 4

Tier2 Center Tier 2

Experiment

CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1

Emerging Emerging Data GridData Grid User Communities User Communities

Grid Physics Network (GriPhyN)Grid Physics Network (GriPhyN) ATLAS, CMS, LIGO, SDSSATLAS, CMS, LIGO, SDSS

Particle Physics Data Grid (PPDG)Particle Physics Data Grid (PPDG)Int’l Virtual Data Grid Lab (iVDGL)Int’l Virtual Data Grid Lab (iVDGL)

NSF Network for Earthquake Engineering NSF Network for Earthquake Engineering Simulation (NEES)Simulation (NEES) Integrated instrumentation, collaboration, Integrated instrumentation, collaboration,

simulationsimulation Access Grid; VRVS: supporting Access Grid; VRVS: supporting

group-based collaborationgroup-based collaboration

AndAnd Genomics, Proteomics, ...Genomics, Proteomics, ... The Earth System Grid and EOSDISThe Earth System Grid and EOSDIS Federating Brain DataFederating Brain Data Computed MicroTomography …Computed MicroTomography … Virtual Observatories Virtual Observatories

HENP Related Data Grid HENP Related Data Grid ProjectsProjects

ProjectsProjects PPDG IPPDG I USAUSA DOEDOE $2M$2M 1999-2001 1999-2001 GriPhyNGriPhyN USAUSA NSF $11.9M + $1.6M 2000-2005NSF $11.9M + $1.6M 2000-2005 EU DataGridEU DataGrid EUEU ECEC €10M€10M 2001-2004 2001-2004 PPDG II (CP) USAPPDG II (CP) USA DOEDOE $9.5M$9.5M 2001-2004 2001-2004 iVDGLiVDGL USAUSA NSFNSF $13.7M + $2M$13.7M + $2M 2001-2006 2001-2006 DataTAGDataTAG EUEU ECEC €4M €4M 2002-2004 2002-2004 GridPP GridPP UKUK PPARCPPARC >$15M>$15M 2001-2004 2001-2004 LCG (Ph1)LCG (Ph1) CERN MSCERN MS 30 MCHF30 MCHF 2002-2004 2002-2004

Many Other Projects of interest to HENPMany Other Projects of interest to HENP Initiatives in US, UK, Italy, France, NL, Germany, Japan, …Initiatives in US, UK, Italy, France, NL, Germany, Japan, … Networking initiatives: Networking initiatives: DataTAG, AMPATH, CALREN-XD… DataTAG, AMPATH, CALREN-XD… US Distributed Terascale Facility: US Distributed Terascale Facility:

($53M, 12 TeraFlops, 40 Gb/s network)($53M, 12 TeraFlops, 40 Gb/s network)

Daily, Weekly, Monthly and Yearly Statistics on 155 Mbps US-CERN LinkDaily, Weekly, Monthly and Yearly

Statistics on 155 Mbps US-CERN Link

20 - 100 Mbps Used Routinely in ’01BaBar: 600 Mbps Throughput in ‘02

BW Upgrades Quickly Followedby Upgraded Production Use

Tier A

"Physicists have indeed foreseen to test the GRID principles starting first from the Computing Centres in Lyon and Stanford (California). A first step towards the ubiquity of the GRID."

Pierre Le HirLe Monde 12 april 2001

CERN-US Line + Abilene

Renater + ESnet3/2002

D. Linglin: LCG Wkshop

Two centers are trying to work as one:-Data not duplicated-Internationalization-transparent access, etc…

RNP Brazil (to 20 Mbps)

FIU Miami/So. America (to 80 Mbps)

Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*]

Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*]

2001 2002 2003 2004 2005 2006

CMS 100 200 300 600 800 2500

ATLAS 50 100 300 600 800 2500

BaBar 300 600 1100 1600 2300 3000

CDF 100 300 400 2000 3000 6000

D0 400 1600 2400 3200 6400 8000

BTeV 20 40 100 200 300 500

DESY 100 180 210 240 270 300

CERN BW

155-310

622 2500 5000 10000 20000

[*] [*] Installed BW. Maximum Link Occupancy 50% Installed BW. Maximum Link Occupancy 50%

AssumedAssumed

See http://gate.hep.anl.gov/lprice/TANSee http://gate.hep.anl.gov/lprice/TAN

MONARC: CMS Analysis ProcessMONARC: CMS Analysis Process

Hierarchy of Processes (Experiment, Analysis Groups,Individuals)Hierarchy of Processes (Experiment, Analysis Groups,Individuals)

SelectionSelection

Iterative selectionIterative selectionOnce per monthOnce per month

~20 Groups’~20 Groups’ActivityActivity

(10(109 9 101077 events) events)

Trigger based andTrigger based andPhysics basedPhysics basedrefinementsrefinements

25 25 SI95sec/eventSI95sec/event~20 jobs per ~20 jobs per

monthmonth

25 25 SI95sec/eventSI95sec/event~20 jobs per ~20 jobs per

monthmonth

AnalysisAnalysisDifferent Physics cutsDifferent Physics cuts

& MC comparison& MC comparison~Once per day~Once per day

~25 Individual~25 Individualper Groupper GroupActivityActivity

(10(1066 –10 –1077 events) events)

Algorithms Algorithms applied to applied to

datadatato get to get resultsresults

10 SI95sec/event10 SI95sec/event~500 jobs per ~500 jobs per

dayday

10 SI95sec/event10 SI95sec/event~500 jobs per ~500 jobs per

dayday

Monte CarloMonte Carlo

5000 5000 SI95sec/eventSI95sec/event

5000 5000 SI95sec/eventSI95sec/event

RAW DataRAW Data

ReconstructionReconstruction Re-processingRe-processing3 Times per year3 Times per year

Experiment-Experiment-Wide ActivityWide Activity(10(1099 events) events)

New detector New detector calibrationscalibrations

Or understandingOr understanding

3000 3000 SI95sec/eventSI95sec/event

1 job year1 job year

3000 3000 SI95sec/eventSI95sec/event

1 job year1 job year

3000 3000 SI95sec/eventSI95sec/event3 jobs per year3 jobs per year

3000 3000 SI95sec/eventSI95sec/event3 jobs per year3 jobs per year

Tier0-Tier1 Link RequirementsTier0-Tier1 Link Requirements Estimate: for Hoffmann Report 2001 Estimate: for Hoffmann Report 2001

1) 1) Tier1 Tier1 Tier0 Data Flow for Analysis Tier0 Data Flow for Analysis 0.5 - 1.0 Gbps0.5 - 1.0 Gbps

2) Tier2 2) Tier2 Tier0 Data Flow for Analysis Tier0 Data Flow for Analysis 0.2 - 0.5 Gbps0.2 - 0.5 Gbps

3) Interactive Collaborative Sessions (30 Peak) 3) Interactive Collaborative Sessions (30 Peak) 0.1 - 0.3 Gbps0.1 - 0.3 Gbps

4) Remote Interactive Sessions (30 Flows Peak) 4) Remote Interactive Sessions (30 Flows Peak) 0.1 - 0.2 Gbps0.1 - 0.2 Gbps

5) Individual (Tier3 or Tier4) data transfers 5) Individual (Tier3 or Tier4) data transfers 0.8 Gbps0.8 GbpsLimit to 10 Flows of 5 Mbytes/sec eachLimit to 10 Flows of 5 Mbytes/sec each

TOTAL Per Tier0 - Tier1 LinkTOTAL Per Tier0 - Tier1 Link 1.7 - 2.8 Gbps1.7 - 2.8 Gbps

NOTE:NOTE: Adopted by the LHC Experiments; given in the Steering Committee Adopted by the LHC Experiments; given in the Steering Committee

Report on LHC Computing: “1.5 - 3 Gbps per experiment”Report on LHC Computing: “1.5 - 3 Gbps per experiment” Corresponds to ~10 Gbps Baseline BW Installed Corresponds to ~10 Gbps Baseline BW Installed

on US-CERN Linkon US-CERN Link Report also discussed the effects of higher bandwidthsReport also discussed the effects of higher bandwidths

For example all-optical 10 Gbps Ethernet + WAN by 2002-3For example all-optical 10 Gbps Ethernet + WAN by 2002-3

Tier0-Tier1 BW RequirementsEstimate: for Hoffmann Report 2001 Tier0-Tier1 BW RequirementsEstimate: for Hoffmann Report 2001

Does Not IncludeDoes Not Include more recent ATLAS Data Estimates more recent ATLAS Data Estimates 270 Hz at 10270 Hz at 103333 Instead of 100Hz Instead of 100Hz 400 Hz at 10400 Hz at 103434 Instead of 100Hz Instead of 100Hz 2 MB/Event Instead of 1 MB/Event ?2 MB/Event Instead of 1 MB/Event ?

Does Not Allow Fast Download to Tier3+4 Does Not Allow Fast Download to Tier3+4 of “Small” Object Collectionsof “Small” Object Collections Example: Download 10Example: Download 1077 Events of AODs (10 Events of AODs (104 4 Bytes) Bytes) 100 100

Gbytes; At 5 Mbytes/sec per person (above) that’s 6 Hours !Gbytes; At 5 Mbytes/sec per person (above) that’s 6 Hours ! This is a still a rough, bottoms-up, static, and hence This is a still a rough, bottoms-up, static, and hence

Conservative Model. Conservative Model. A Dynamic distributed DB or “Grid” system with Caching, A Dynamic distributed DB or “Grid” system with Caching,

Co-scheduling, and Pre-Emptive data movement Co-scheduling, and Pre-Emptive data movement may well require greater bandwidthmay well require greater bandwidth

Does Not Include “Virtual Data” operations;Does Not Include “Virtual Data” operations;Derived Data Copies; Data-description overheadsDerived Data Copies; Data-description overheads

Further MONARC Model Studies are NeededFurther MONARC Model Studies are Needed

*

Also see http://www-iepm.slac.stanford.edu/monitoring/bulk/; and the Internet2 E2E Initiative: http://www.internet2.edu/e2e

Maximum Throughput on Transatlantic Links (155 Mbps)

Maximum Throughput on Transatlantic Links (155 Mbps)

8/10/01 105 Mbps reached with 30 Streams: SLAC-IN2P38/10/01 105 Mbps reached with 30 Streams: SLAC-IN2P3 9/1/01 102 Mbps in One Stream: CIT-CERN9/1/01 102 Mbps in One Stream: CIT-CERN 11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN11/5/01 125 Mbps in One Stream (modified kernel): CIT-CERN 1/09/02 190 Mbps for One stream shared on 2 155 Mbps links1/09/02 190 Mbps for One stream shared on 2 155 Mbps links 3/11/02 120 Mbps 3/11/02 120 Mbps Disk-to-DiskDisk-to-Disk with One Stream on 155 Mbps with One Stream on 155 Mbps

link (Chicago-CERN)link (Chicago-CERN) 5/20/02 450 Mbps SLAC-Manchester on OC12 with ~100 Streams5/20/02 450 Mbps SLAC-Manchester on OC12 with ~100 Streams 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel) 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 (mod. Kernel)

Some Recent Events:Reported 6/1/02 to ICFA/SCIC

Some Recent Events:Reported 6/1/02 to ICFA/SCIC

Progress in High Throughput: 0.1 to 1 GbpsProgress in High Throughput: 0.1 to 1 Gbps Land Speed Record: SURFNet – Alaska (IPv6) Land Speed Record: SURFNet – Alaska (IPv6)

(0.4+ Gbps)(0.4+ Gbps) SLAC – Manchester (Les C. and Richard H-J) SLAC – Manchester (Les C. and Richard H-J)

(0.4+ Gbps)(0.4+ Gbps) Tsunami (Indiana) (0.8 Gbps UDP)Tsunami (Indiana) (0.8 Gbps UDP) Tokyo – KEK (0.5 – 0.9 Gbps)Tokyo – KEK (0.5 – 0.9 Gbps)

Progress in Pre-Production and Production NetworkingProgress in Pre-Production and Production Networking 10 Mbytes/sec FNAL-CERN (Michael Ernst)10 Mbytes/sec FNAL-CERN (Michael Ernst) 15 Mbytes/sec disk-to-disk Chicago-CERN 15 Mbytes/sec disk-to-disk Chicago-CERN

(Sylvain Ravot)(Sylvain Ravot) KPNQwest files for Chapter 11; Stops network yesterday.KPNQwest files for Chapter 11; Stops network yesterday.

Near Term Pricing of Competitor (DT) ok.Near Term Pricing of Competitor (DT) ok. Unknown impact on prices and future planning Unknown impact on prices and future planning

in the medium and longer termin the medium and longer term

Baseline BW for the US-CERN Link:Baseline BW for the US-CERN Link: HENP Transatlantic WG (DOE+NSF HENP Transatlantic WG (DOE+NSF))

US-CERN Link: 622 Mbps this monthUS-CERN Link: 622 Mbps this month DataTAG 2.5 Gbps Research Link in Summer 2002DataTAG 2.5 Gbps Research Link in Summer 2002 10 Gbps Research Link by Approx. Mid-200310 Gbps Research Link by Approx. Mid-2003

Transoceanic Networking

Integrated with the Abilene,

TeraGrid, Regional Nets

and Continental Network

Infrastructuresin US, Europe,

Asia, South America

Baseline evolution typicalBaseline evolution typicalof major HENPof major HENP

links 2001-2006 links 2001-2006

Total U.S. Internet TrafficTotal U.S. Internet Traffic

Source: Roberts et al., 2001

U.S. Internet TrafficU.S. Internet Traffic

1970 1975 1980 1985 1990 1995 2000 2005 2010

Voice Crossover: August 2000

4X/Year2.8X/Year

1Gbps

1Tbps10Tbps

100Gbps10Gbps

100Tbps

100Mbps

1Kbps

1Mbps10Mbps

100Kbps10Kbps

100 bps

1 Pbps

100 Pbps10 Pbps

10 bps

ARPA & NSF Data to 96

New Measurements

Limit of same % GDP as Voice

Projected at 3/Year

Internet Growth Rate Fluctuates Over TimeInternet Growth Rate Fluctuates Over Time

U.S. Internet Edge Traffic Growth Rate6 Month Lagging Measure

0.000.00

0.500.50

1.001.00

1.501.50

2.002.00

2.502.50

3.003.00

3.503.50

4.004.00

4.504.50

Jan 00Jan 00 Apr 00Apr 00 Jul 00Jul 00 Oct 00Oct 00 Jan 01Jan 01 Apr 01Apr 01 Jul 01Jul 01 Oct 01Oct 01 Jan 02Jan 02

Gro

wth

Rat

e p

er Y

ear

Gro

wth

Rat

e p

er Y

ear

Average: 3.0/yearAverage: 3.0/year

10/00–4/01 Growth 10/00–4/01 Growth Reported 3.6/yearReported 3.6/year 10/00–4/01 Growth 10/00–4/01 Growth

Reported 4.0/yearReported 4.0/year

Source: Roberts et al., 2002

AMS-IX Internet Exchange Throughput Accelerating Growth in Europe (NL)

AMS-IX Internet Exchange Throughput Accelerating Growth in Europe (NL)

Monthly Traffic2X Growth from 8/00 - 3/01;2X Growth from 8/01 - 12/01 ↓

2.0 Gbps

4.0 Gbps

6.0 GbpsHourly Traffic

3/22/02

ICFA SCIC Meeting March 9 at CERN: Updates from Members

ICFA SCIC Meeting March 9 at CERN: Updates from Members

Abilene UpgradeAbilene Upgrade from 2.5 to 10 Gbps from 2.5 to 10 Gbps Additional scheduled lambdas planned for targeted Additional scheduled lambdas planned for targeted

for targeted applications: Pacific and National Light Rail for targeted applications: Pacific and National Light Rail US-CERNUS-CERN

Upgrade On Track: to 622 Mbps in July; Upgrade On Track: to 622 Mbps in July; Setup and Testing Done in STARLIGHTSetup and Testing Done in STARLIGHT

2.5G Research Lambda by this Summer: STARLIGHT-CERN2.5G Research Lambda by this Summer: STARLIGHT-CERN 2.5G Triangle between STARLIGHT (US), SURFNet (NL), 2.5G Triangle between STARLIGHT (US), SURFNet (NL),

CERNCERN SLAC + IN2P3 (BaBar) SLAC + IN2P3 (BaBar)

Getting 100 Mbps over 155 Mbps CERN-US LinkGetting 100 Mbps over 155 Mbps CERN-US Link 50 Mbps Over RENATER 155 Mbps Link, Limited by ESnet50 Mbps Over RENATER 155 Mbps Link, Limited by ESnet 600 Mbps Throughput is BaBar Target for this Year600 Mbps Throughput is BaBar Target for this Year

FNALFNAL Expect ESnet Upgrade to 622 Mbps this MonthExpect ESnet Upgrade to 622 Mbps this Month Plans for dark fiber to STARLIGHT underway, could be Plans for dark fiber to STARLIGHT underway, could be

done in ~4 Months; Railway or Electric Co. providerdone in ~4 Months; Railway or Electric Co. provider

ICFA SCIC: A&R Backbone and International Link Progress

ICFA SCIC: A&R Backbone and International Link Progress

GEANT Pan-European BackboneGEANT Pan-European Backbone ( (http://www.dante.net/geanthttp://www.dante.net/geant)) Now interconnects 31 countriesNow interconnects 31 countries Includes many trunks at 2.5 and 10 GbpsIncludes many trunks at 2.5 and 10 Gbps

UKUK 2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene2.5 Gbps NY-London, with 622 Mbps to ESnet and Abilene

SuperSINET (Japan):SuperSINET (Japan): 10 Gbps IP and 10 Gbps Wavelength 10 Gbps IP and 10 Gbps Wavelength Upgrade to Two 0.6 Gbps Links, to Chicago and SeattleUpgrade to Two 0.6 Gbps Links, to Chicago and Seattle Plan upgrade to 2 X 2.5 Gbps Connection to Plan upgrade to 2 X 2.5 Gbps Connection to

US West Coast by 2003US West Coast by 2003 CA*net4 (Canada):CA*net4 (Canada): Interconnect customer-owned dark fiber Interconnect customer-owned dark fiber

nets across Canada at 10 Gbps, starting July 2002 nets across Canada at 10 Gbps, starting July 2002 ““Lambda-Grids” by ~2004-5Lambda-Grids” by ~2004-5

GWIN (Germany):GWIN (Germany): Connection to Abilene Upgraded Connection to Abilene Upgraded to 2 X 2.5 Gbps early in 2002 to 2 X 2.5 Gbps early in 2002

RussiaRussia Start 10 Mbps link to CERN and ~90 Mbps to US NowStart 10 Mbps link to CERN and ~90 Mbps to US Now

210 Primary ParticipantsAll 50 States, D.C. and Puerto Rico80 Partner Corporations and Non-

Profits22 State Research and Education

Nets 15 “GigaPoPs” Support 70% of

Members

2.510 Gbps Backbone

Caltech Connection with GbE to New Backbone

National R&E Network ExampleGermany: DFN TransAtlanticConnectivity Q1 2002

STM 4

STM 16

STM 16

2 X OC12 Now: NY-Hamburg 2 X OC12 Now: NY-Hamburg and NY-Frankfurtand NY-Frankfurt

ESNet peering at 34 MbpsESNet peering at 34 Mbps Upgrade to 2 X OC48 expected Upgrade to 2 X OC48 expected

in Q1 2002 in Q1 2002 Direct Peering to Abilene and Direct Peering to Abilene and

Canarie expectedCanarie expected UCAID will add another 2 OC48’s; UCAID will add another 2 OC48’s;

Proposing a Global Terabit Proposing a Global Terabit Research Network (GTRN) Research Network (GTRN)

FSU Connections via satellite:FSU Connections via satellite:Yerevan, Minsk, Almaty, BaikalYerevan, Minsk, Almaty, Baikal Speeds of 32 - 512 kbpsSpeeds of 32 - 512 kbps

SILK Project (2002): NATO fundingSILK Project (2002): NATO funding Links to Caucasus and CentralLinks to Caucasus and Central Asia (8 Countries) Asia (8 Countries)

Currently 64-512 kbpsCurrently 64-512 kbpsPropose VSAT for 10-50 X BW:Propose VSAT for 10-50 X BW: NATO + State Funding NATO + State Funding

National Research Networks in Japan

National Research Networks in Japan

SuperSINET SuperSINET Started operation January 4, 2002Started operation January 4, 2002 Support for 5 important areas:Support for 5 important areas:

HEP,HEP, Genetics, Nano-Technology, Genetics, Nano-Technology,Space/Astronomy, Space/Astronomy, GRIDsGRIDs

Provides 10 Provides 10 ’s:’s: 10 Gbps IP connection 10 Gbps IP connection 7 Direct intersite GbE links7 Direct intersite GbE links Some connections to Some connections to 10 GbE 10 GbE in JFY2002in JFY2002

HEPnet-J HEPnet-J Will be re-constructed with Will be re-constructed with MPLS-VPN in SuperSINET MPLS-VPN in SuperSINET

Proposal: Two TransPacific Proposal: Two TransPacific 2.5 Gbps Wavelengths, and 2.5 Gbps Wavelengths, and Japan-CERN Grid Testbed by ~2003 Japan-CERN Grid Testbed by ~2003

Tokyo

Osaka

Nagoya

Internet

Osaka U

Kyoto U

ICR

Kyoto-U

Nagoya U

NIFS

NIG

KEK

Tohoku U

IMS

U-TokyoNAO

U Tokyo

NII Hitot.

NII Chiba

IP

WDM path

IP router

OXC

ISAS

NLNLSURFnet

GENEVA

UKUKSuperJANET4

ABILENE

ABILENE

ESNETESNET

CALREN

CALREN

ItItGARR-B

GEANT

NewYork

FrFrRenater

STAR-TAP

STARLIGHT

DataTAG ProjectDataTAG Project

EU-Solicited Project. EU-Solicited Project. CERNCERN, PPARC (UK), Amsterdam (NL), and INFN (IT);, PPARC (UK), Amsterdam (NL), and INFN (IT);and US (DOE/NSF: UIC, NWU and Caltech) partnersand US (DOE/NSF: UIC, NWU and Caltech) partners

Main Aims: Main Aims: Ensure maximum interoperability between US and EU Grid ProjectsEnsure maximum interoperability between US and EU Grid ProjectsTransatlantic Testbed for advanced network researchTransatlantic Testbed for advanced network research

2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle in 2003)2.5 Gbps Wavelength Triangle 7/02 (10 Gbps Triangle in 2003)

Wave

Triangle

TeraGrid (www.teragrid.org)TeraGrid (www.teragrid.org)NCSA, ANL, SDSC, CaltechNCSA, ANL, SDSC, Caltech

NCSA/UIUC

ANL

UIC Multiple Carrier Hubs

Starlight / NW UnivStarlight / NW Univ

Ill Inst of Tech

Univ of ChicagoIndianapolis (Abilene NOC)

I-WIRE

Caltech

San Diego

DTF Backplane: 4 X 10 Gbps

AbileneChicago

Indianapolis

Urbana

OC-48 (2.5 Gb/s, Abilene)Multiple 10 GbE (Qwest)Multiple 10 GbE (I-WIRE Dark Fiber)

Source: Charlie Catlett, Argonne

A Preview of the Grid Hierarchyand Networks of the LHC Era

Idea to extend the TeraGrid to CERN

CA ONI, CALREN-XD + Pacific Light CA ONI, CALREN-XD + Pacific Light Rail Backbones (Proposed)Rail Backbones (Proposed)

Also: LA-CaltechMetro Fiber;

National Light Rail

Key Network Issues & Challenges

Key Network Issues & Challenges

Net Infrastructure Requirements for High ThroughputNet Infrastructure Requirements for High Throughput Packet Loss must be ~Zero (at and below 10Packet Loss must be ~Zero (at and below 10-6-6))

I.e. No “Commodity” networksI.e. No “Commodity” networks Need to track down uncongested packet lossNeed to track down uncongested packet loss

No Local infrastructure bottlenecksNo Local infrastructure bottlenecks Multiple Gigabit Ethernet “clear paths” between Multiple Gigabit Ethernet “clear paths” between

selected host pairs are needed nowselected host pairs are needed now To 10 Gbps Ethernet paths by 2003 or 2004To 10 Gbps Ethernet paths by 2003 or 2004

TCP/IP stack configuration and tuning Absolutely RequiredTCP/IP stack configuration and tuning Absolutely Required Large Windows; Possibly Multiple StreamsLarge Windows; Possibly Multiple Streams New Concepts of New Concepts of Fair UseFair Use Must then be Developed Must then be Developed

Careful Router, Server, Client, Interface configuration Careful Router, Server, Client, Interface configuration Sufficient CPU, I/O and NIC throughput sufficientSufficient CPU, I/O and NIC throughput sufficient

End-to-endEnd-to-end monitoring and tracking of performance monitoring and tracking of performance Close collaboration with local and “regional” network staffsClose collaboration with local and “regional” network staffs

TCP Does Not Scale to the 1-10 Gbps RangeTCP Does Not Scale to the 1-10 Gbps Range

A Short List: Revolutions in Information Technology (2002-7)

A Short List: Revolutions in Information Technology (2002-7)

Managed Managed Global Data Grids (As Above)Global Data Grids (As Above) Scalable Data-Intensive Metro and Long HaulScalable Data-Intensive Metro and Long Haul

Network TechnologiesNetwork Technologies DWDM: 10 Gbps then 40 Gbps per DWDM: 10 Gbps then 40 Gbps per ; ;

1 to 10 Terabits/sec per fiber 1 to 10 Terabits/sec per fiber 10 Gigabit Ethernet (See www.10gea.org) 10 Gigabit Ethernet (See www.10gea.org)

10GbE / 10 Gbps LAN/WAN integration 10GbE / 10 Gbps LAN/WAN integration Metro Buildout and Optical Cross ConnectsMetro Buildout and Optical Cross Connects Dynamic Provisioning Dynamic Provisioning Dynamic Path Building Dynamic Path Building

““Lambda GridsLambda Grids” ” Defeating the “Last Mile” ProblemDefeating the “Last Mile” Problem

(Wireless; or Ethernet in the First Mile) (Wireless; or Ethernet in the First Mile) 3G and 4G Wireless Broadband (from ca. 2003);3G and 4G Wireless Broadband (from ca. 2003);

and/or Fixed Wireless “Hotspots” and/or Fixed Wireless “Hotspots” Fiber to the HomeFiber to the Home Community-Owned NetworksCommunity-Owned Networks

A Short List: Coming Revolutions in Information Technology

A Short List: Coming Revolutions in Information Technology

Storage VirtualizationStorage Virtualization Grid-enabled Storage Resource Middleware (SRM)Grid-enabled Storage Resource Middleware (SRM) iSCSI (Internet Small Computer Storage Interface);iSCSI (Internet Small Computer Storage Interface);

Integrated with 10 GbE Integrated with 10 GbE Global File Systems Global File Systems Internet Information Software TechnologiesInternet Information Software Technologies

Global Information “Broadcast” ArchitectureGlobal Information “Broadcast” ArchitectureE.g the Multipoint Information Distribution E.g the Multipoint Information Distribution

Protocol (Tie.Liao@inria.fr)Protocol (Tie.Liao@inria.fr) Programmable Coordinated Agent ArchitecturesProgrammable Coordinated Agent Architectures

E.g. Mobile Agent Reactive Spaces (MARS)E.g. Mobile Agent Reactive Spaces (MARS) by Cabri et al., University of Modena by Cabri et al., University of Modena

The “Data Grid” - Human InterfaceThe “Data Grid” - Human Interface Interactive monitoring and control of Grid resources Interactive monitoring and control of Grid resources

By authorized groups and individualsBy authorized groups and individualsBy Autonomous AgentsBy Autonomous Agents

HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps

HENP Major Links: Bandwidth Roadmap (Scenario) in Gbps

Year Production Experimental Remarks

2001 0.155 0.622-2.5 SONET/SDH

2002 0.622 2.5 SONET/SDH DWDM; GigE Integ.

2003 2.5 10 DWDM; 1 + 10 GigE Integration

2005 10 2-4 X 10 Switch; Provisioning

2007 2-4 X 10 ~10 X 10; 40 Gbps

1st Gen. Grids

2008 ~10 X 10 or 1-2 X 40

~5 X 40 or ~20-50 X 10

40 Gbps Switching

2010 ~5 X 40 or

~20 X 10

~25 X 40 or ~100 X 10

2nd Gen Grids Terabit Networks

2012 ~Terabit ~MultiTerabit ~Fill One Fiber or Use a Few Fibers

One Long Range Scenario (Ca. 2008-12)HENP As a Driver of Optical NetworksPetascale Grids with TB Transactions

One Long Range Scenario (Ca. 2008-12)HENP As a Driver of Optical NetworksPetascale Grids with TB Transactions

Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Storesfrom 1 to 1000 Petabyte Data Stores

Survivability of the HENP Global Grid System, with Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007)hundreds of such transactions per day (circa 2007)requires that each transaction be completed in a requires that each transaction be completed in a relatively short time. relatively short time.

Example: Take 800 secs to complete the transaction. ThenExample: Take 800 secs to complete the transaction. Then Transaction Size (TB)Transaction Size (TB) Net Throughput (Gbps)Net Throughput (Gbps) 1 101 10 10 10010 100 100 1000 (Capacity of 100 1000 (Capacity of

Fiber Today) Fiber Today) Summary: Providing Switching of 10 Gbps wavelengthsSummary: Providing Switching of 10 Gbps wavelengths

within ~3 years; and Terabit Switching within 5-10 yearswithin ~3 years; and Terabit Switching within 5-10 yearswould enable “Petascale Grids with Terabyte transactions”,would enable “Petascale Grids with Terabyte transactions”,as required to fully realize the discovery potential of major as required to fully realize the discovery potential of major HENP programs, as well as other data-intensive fields.HENP programs, as well as other data-intensive fields.

Internet2 HENP WG [*]Internet2 HENP WG [*] Mission: To help ensure that the requiredMission: To help ensure that the required

National and international network infrastructuresNational and international network infrastructures(end-to-end)(end-to-end)

Standardized tools and facilities for high performance Standardized tools and facilities for high performance and end-to-end monitoring and tracking, andand end-to-end monitoring and tracking, and

Collaborative systemsCollaborative systems are developed and deployed in a timely manner, and used are developed and deployed in a timely manner, and used

effectively to meet the needs of the US LHC and other major effectively to meet the needs of the US LHC and other major HENP Programs, as well as the at-large scientific community.HENP Programs, as well as the at-large scientific community. To carry out these developments in a way that is To carry out these developments in a way that is

broadly applicable across many fields broadly applicable across many fields Formed an Internet2 WG as a suitable framework: Formed an Internet2 WG as a suitable framework:

Oct. 26 2001 Oct. 26 2001 [*] Co-Chairs: S. McKee (Michigan), H. Newman (Caltech);[*] Co-Chairs: S. McKee (Michigan), H. Newman (Caltech);

Sec’y J. Williams (Indiana) Sec’y J. Williams (Indiana) Website: Website: http://www.internet2.edu/henphttp://www.internet2.edu/henp; also see the Internet2; also see the Internet2

End-to-end Initiative: End-to-end Initiative: http://www.internet2.edu/e2ehttp://www.internet2.edu/e2e

True End to End Experience

User perception ApplicationOperating systemHost IP stackHost network cardLocal Area Network Campus backbone

networkCampus link to regional

network/GigaPoPGigaPoP link to

Internet2 national backbones

International connections

EYEBALL

APPLICATION

STACK

JACK

NETWORK

. . .

. . .

. . .

. . .

HENP Scenario Limitations:Technologies and Costs

HENP Scenario Limitations:Technologies and Costs

Router Technology and Costs Router Technology and Costs (Ports and Backplane) (Ports and Backplane)

Computer CPU, Disk and I/O ChannelComputer CPU, Disk and I/O Channel Speeds to Send and Receive Data Speeds to Send and Receive Data

Link Costs: Unless Dark Fiber (?)Link Costs: Unless Dark Fiber (?) MultiGigabit Transmission Protocols MultiGigabit Transmission Protocols

End-to-End End-to-End ““100 GbE” Ethernet (or something else) by100 GbE” Ethernet (or something else) by

~2006: for LANs to match WAN speeds ~2006: for LANs to match WAN speeds

[*] See “Macroscopic Behavior of the TCP Congestion Avoidance Algorithm,” Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), 7/1997

Throughput quality improvements:BWTCP < MSS/(RTT*sqrt(loss)) [*]

Throughput quality improvements:BWTCP < MSS/(RTT*sqrt(loss)) [*]

China Improves But Far Behind

80% Improvement/Year

Factor of 10 In 4 Years

Eastern Europe Far Behind

11900 Hosts;6620 Registered Users in 61 Countries 43 (7 I2) Reflectors Annual Growth 2 to 3X

Networks, Grids and HENPNetworks, Grids and HENP Next generation 10 Gbps network backbones are Next generation 10 Gbps network backbones are

almost here: in the US, Europe and Japanalmost here: in the US, Europe and Japan First stages arriving, starting nowFirst stages arriving, starting now

Major transoceanic links at 2.5 - 10 Gbps in 2002-3Major transoceanic links at 2.5 - 10 Gbps in 2002-3 Network improvements are especially needed in Network improvements are especially needed in

Southeast Europe, So. America; and some other regions:Southeast Europe, So. America; and some other regions: Romania, Brazil; India, Pakistan, China; Africa Romania, Brazil; India, Pakistan, China; Africa

Removing regional, last mile bottlenecks and Removing regional, last mile bottlenecks and compromises in network quality are now compromises in network quality are now All on the critical pathAll on the critical path

Getting high (reliable; Grid) application performance Getting high (reliable; Grid) application performance across networks means!across networks means!

End-to-end monitoring; a coherent approach End-to-end monitoring; a coherent approach Getting high performance (TCP) toolkits in users’ Getting high performance (TCP) toolkits in users’

handshands Working in concert with AMPATH, Internet E2E, I2 Working in concert with AMPATH, Internet E2E, I2

HENP WG, DataTAG; the Grid projects and the GGFHENP WG, DataTAG; the Grid projects and the GGF