HENP HENP Networks and Grids for Global ScienceNetworks and Grids for Global Science

Harvey B. NewmanHarvey B. Newman California Institute of TechnologyCalifornia Institute of Technology33rdrd International Data Grid Workshop International Data Grid Workshop

Daegu, Korea August 26, 2004Daegu, Korea August 26, 2004

Challenges for Global HENP ExperimentsChallenges for Global HENP Experiments

Major Challenges (Shared with Other Fields)Major Challenges (Shared with Other Fields) Worldwide Communication and Collaboration Worldwide Communication and Collaboration Managing Globally Distributed Computing & Data Resources Managing Globally Distributed Computing & Data Resources Cooperative Software Development and Data AnalysisCooperative Software Development and Data Analysis

LHC Example- 20075000+ Physicists 250+ Institutes 60+ Countries

BaBar/D0 Example - 2004500+ Physicists 100+ Institutes 35+ Countries

First Beams: Summer 2007Physics Runs: from Fall 2007

TOTEM pp, general purpose; HI

pp, general purpose; HI

LHCb: B-physics

ALICE : HI

pp s =14 TeV L=1034 cm-2 s-1

27 km Tunnel in Switzerland & France

Large Hadron Collider (LHC) CERN, Geneva: 2007 Start

Large Hadron Collider (LHC) CERN, Geneva: 2007 Start

CMS

Atlas

Higgs, SUSY, QG Plasma, CP Violation, … the Unexpected

Challenges of Next Generation Science in the Information AgeChallenges of Next Generation Science in the Information Age

Flagship Applications Flagship Applications High Energy & Nuclear Physics, AstroPhysics Sky Surveys:High Energy & Nuclear Physics, AstroPhysics Sky Surveys:

TByte to PByte “block” transfers at 1-10+ Gbps TByte to PByte “block” transfers at 1-10+ Gbps Fusion Energy:Fusion Energy: Time Critical Burst-Data Distribution; Time Critical Burst-Data Distribution;

Distributed Plasma Simulations, Visualization and Analysis; Distributed Plasma Simulations, Visualization and Analysis; Preparations for Fusion Energy Experiment Preparations for Fusion Energy Experiment

eVLBI:eVLBI: Many real time data streams at 1-10 Gbps Many real time data streams at 1-10 Gbps BioInformatics, Clinical Imaging:BioInformatics, Clinical Imaging: GByte images on demand GByte images on demand

Provide results with rapid turnaround, coordinating Provide results with rapid turnaround, coordinating large but limited computing and data handling resources,large but limited computing and data handling resources,over networks of varying capability in different world regionsover networks of varying capability in different world regions

Advanced integrated applications, such as Data Grids, Advanced integrated applications, such as Data Grids, rely on seamless operation of our LANs and WANsrely on seamless operation of our LANs and WANs With reliable, quantifiable high performanceWith reliable, quantifiable high performance

Petabytes of complex data explored and analyzed by Petabytes of complex data explored and analyzed by 1000s of globally dispersed scientists, in hundreds of teams1000s of globally dispersed scientists, in hundreds of teams

LHC Data Grid Hierarchy:Developed at Caltech

LHC Data Grid Hierarchy:Developed at Caltech

Tier 1

Tier2 Center

Online System

CERN Center PBs of Disk;

Tape Robot

FNAL CenterIN2P3 Center INFN Center RAL Center

InstituteInstituteInstituteInstitute

Workstations

~100-1500 MBytes/sec

~10 Gbps

1 to 10 Gbps

Tens of Petabytes by 2007-8.An Exabyte ~5-7 Years later.Physics data cache

~PByte/sec

10 - 40 Gbps

Tier2 CenterTier2 CenterTier2 Center

~10 Gbps

Tier 0 +1

Tier 3

Tier 4

Tier2 Center Tier 2

Experiment

CERN/Outside Resource Ratio ~1:2Tier0/( Tier1)/( Tier2) ~1:1:1

Emerging Vision: A Richly Structured, Global Dynamic System

ICFA and Global Networks ICFA and Global Networks for Collaborative Sciencefor Collaborative Science

ICFA and Global Networks ICFA and Global Networks for Collaborative Sciencefor Collaborative Science

National and International Networks, with sufficient National and International Networks, with sufficient (rapidly increasing) capacity and seamless end-to-end (rapidly increasing) capacity and seamless end-to-end capability, are essential forcapability, are essential forThe daily conduct of collaborative work in bothThe daily conduct of collaborative work in both

experiment and theory experiment and theoryExperiment development & construction Experiment development & construction

on a global scale on a global scaleGrid systems supporting analysis involving Grid systems supporting analysis involving

physicists in all world regions physicists in all world regions The conception, design and implementation of The conception, design and implementation of

next generation facilities as “global networks” next generation facilities as “global networks” ““Collaborations on this scale would never have been Collaborations on this scale would never have been

attempted, if they could not rely on excellent networks”attempted, if they could not rely on excellent networks”

History of Bandwidth Usage – One Large Network; One Large Research Site

History of Bandwidth Usage – One Large Network; One Large Research Site

ESnet Monthly Accepted Traffic 1/90-1/04

0

50

100

150

200

250

300

Jan,

90Oc

t, 90

Jul,9

1Ap

r, 92

Jan,

93Oc

t,93

Jul, 9

4Ap

r, 95

Jan,

96Oc

t, 96

Jul, 9

7Ap

r, 98

Jan,

99Oc

t, 99

Jul, 0

0Ap

r, 01

Jan,

02Oc

t, 02

Jul, 0

3

TByte

/Mon

th

SLAC Traffic ~300 Mbps; ESnet Limit

Growth in Steps: ~ 10X/4 YearsProjected: ~2 Terabits/s by

~2014

ESnet Accepted Traffic 1/90 – 1/04Exponential Growth Since ’92;

Annual Rate Increased from 1.7 to 2.0X Per Year In the Last 5 Years

Int’l Networks BW on Major Links for HENP: US-CERN Example

Int’l Networks BW on Major Links for HENP: US-CERN Example

Rate of Progress >> Moore’s Law (US-CERN Example)Rate of Progress >> Moore’s Law (US-CERN Example) 9.6 kbps Analog9.6 kbps Analog (1985) (1985) 64-256 kbps Digital (1989 - 1994) [X 7 – 27]64-256 kbps Digital (1989 - 1994) [X 7 – 27] 1.5 Mbps Shared (1990-3; IBM) [X 160]1.5 Mbps Shared (1990-3; IBM) [X 160] 2 -4 Mbps2 -4 Mbps (1996-1998) [X 200-400] (1996-1998) [X 200-400] 12-20 Mbps (1999-2000) [X 1.2k-2k]12-20 Mbps (1999-2000) [X 1.2k-2k] 155-310 Mbps155-310 Mbps (2001-2) [X 16k – 32k] (2001-2) [X 16k – 32k] 622 Mbps622 Mbps (2002-3) [X 65k] (2002-3) [X 65k] 2.5 Gbps 2.5 Gbps (2003-4) [X 250k] (2003-4) [X 250k] 10 Gbps 10 Gbps (2005) [X 1M] (2005) [X 1M] 44x10 Gbps or 40 Gbps (2007-8)x10 Gbps or 40 Gbps (2007-8) [X 4M][X 4M]

A factor of ~1M Bandwidth Improvement over A factor of ~1M Bandwidth Improvement over 1985-2005 (a factor of ~5k during 1995-2005)1985-2005 (a factor of ~5k during 1995-2005)

A prime enabler of major HENP programsA prime enabler of major HENP programs HENP has become a leading applications driver, HENP has become a leading applications driver,

and also a co-developer of global networks and also a co-developer of global networks

Internet Growth in the World At Large

Internet Growth in the World At Large

Amsterdam Internet Exchange Point Example

20 Gbps

30 Gbps

Average

5 Minute

Max

Some Annual Growth Spurts;

Typically In Summer-Fall

The Rate of HENP Network Usage Growth (~100% Per Year) is Similar to the World at Large

11.08.04

Judged on product of transfer speed Judged on product of transfer speed and distance end-to-end, using and distance end-to-end, using standard Internet (TCP/IP) protocols.standard Internet (TCP/IP) protocols.

IPv6 record: IPv6 record: 4.0 Gbps4.0 Gbps between between Geneva and Phoenix (SC2003)Geneva and Phoenix (SC2003)

IPv4 Multi-stream record with IPv4 Multi-stream record with Windows & Linux: Windows & Linux: 6.6 Gbps6.6 Gbps between between Caltech and CERN (16 kkm; Caltech and CERN (16 kkm; “Grand Tour d’Abilene”) June 2004“Grand Tour d’Abilene”) June 2004 Exceeded 100 Petabit-m/secExceeded 100 Petabit-m/sec

Single Stream 7.5 Gbps X 16 kkmSingle Stream 7.5 Gbps X 16 kkm with with Linux Achieved in JulyLinux Achieved in July

Concentrate now on reliable Concentrate now on reliable Terabyte-scale file transfersTerabyte-scale file transfersNote System Issues: CPU, PCI-XNote System Issues: CPU, PCI-X

Bus, NIC, I/O Controllers, Drivers Bus, NIC, I/O Controllers, Drivers

Monitoring of the Abilene traffic in LA:Monitoring of the Abilene traffic in LA:

6.6 Gbps16500km

4.2 Gbps16343km5.6 Gbps

10949km

5.4 Gbps7067km

2.5 Gbps10037km0.9 Gbps

10978km0.4 Gbps12272km

0

20

40

60

80

100

120

Internet 2 Land Speed Record (LSR)Internet 2 Land Speed Record (LSR)

Pe

tab

itm

ete

r (1

0^1

5 b

it*m

ete

r)

LSR History – IPv4 single stream

http://www.guinnessworldrecords.com/

June 2004 Record Network

Evolving Quantitative Science Requirements for Networks (DOE High Perf. Network Workshop)

Evolving Quantitative Science Requirements for Networks (DOE High Perf. Network Workshop)

Science AreasScience Areas Today Today End2EndEnd2End

ThroughputThroughput

5 5 years years End2End End2End

ThroughputThroughput

5-10 5-10 Years Years End2End End2End

ThroughputThroughput

RemarksRemarks

High Energy High Energy PhysicsPhysics

0.5 0.5 Gb/sGb/s 100 100 Gb/sGb/s 1000 1000 Gb/sGb/s High bulk High bulk throughputthroughput

Climate (Data & Climate (Data & Computation)Computation)

0.5 0.5 Gb/sGb/s 160-200 160-200 Gb/sGb/s N x 1000 Gb/sN x 1000 Gb/s High bulk High bulk throughputthroughput

SNS NanoScienceSNS NanoScience Not yet startedNot yet started 1 1 Gb/sGb/s 1000 1000 Gb/s + QoS Gb/s + QoS for Control for Control

ChannelChannel

Remote control Remote control and time critical and time critical

throughputthroughput

Fusion EnergyFusion Energy 0.066 0.066 Gb/sGb/s(500 MB/s (500 MB/s

burst)burst)

0.198 0.198 Gb/sGb/s(500MB/(500MB/

20 sec. burst)20 sec. burst)

N x 1000 Gb/sN x 1000 Gb/s Time critical Time critical throughputthroughput

AstrophysicsAstrophysics 0.013 0.013 Gb/sGb/s(1 TByte/week)(1 TByte/week)

N*N multicastN*N multicast 1000 1000 Gb/sGb/s Computational Computational steering and steering and

collaborationscollaborations

Genomics Data & Genomics Data & ComputationComputation

0.091 0.091 Gb/sGb/s(1 TBy/day)(1 TBy/day)

100100s of userss of users 1000 1000 Gb/s + QoS Gb/s + QoS for Control for Control

ChannelChannel

High throughput High throughput and steeringand steering

HENP Lambda Grids:Fibers for Physics

HENP Lambda Grids:Fibers for Physics

Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes Problem: Extract “Small” Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data Storesfrom 1 to 1000 Petabyte Data Stores

Survivability of the HENP Global Grid System, with Survivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007)hundreds of such transactions per day (circa 2007)requires that each transaction be completed in a requires that each transaction be completed in a relatively short time. relatively short time.

Example: Take 800 secs to complete the transaction. ThenExample: Take 800 secs to complete the transaction. Then Transaction Size (TB)Transaction Size (TB) Net Throughput (Gbps)Net Throughput (Gbps) 1 101 10 10 10010 100 100 1000 (Capacity of 100 1000 (Capacity of

Fiber Today) Fiber Today) Summary: Providing Switching of 10 Gbps wavelengthsSummary: Providing Switching of 10 Gbps wavelengths

within ~2-4 years; and Terabit Switching within 5-8 years within ~2-4 years; and Terabit Switching within 5-8 years would enable “Petascale Grids with Terabyte transactions”,would enable “Petascale Grids with Terabyte transactions”,to fully realize the discovery potential of major HENP programs, to fully realize the discovery potential of major HENP programs, as well as other data-intensive research.as well as other data-intensive research.

SCIC in 2003-2004 http://cern.ch/icfa-scic SCIC in 2003-2004 http://cern.ch/icfa-scic

Three 2004 Reports; Presented to ICFA in FebruaryThree 2004 Reports; Presented to ICFA in February

Main Report: “Networking for HENP”Main Report: “Networking for HENP” [H. Newman et al.] [H. Newman et al.] Includes Brief Updates on Monitoring, the Digital Divide Includes Brief Updates on Monitoring, the Digital Divide

and Advanced Technologies [*]and Advanced Technologies [*] A World Network Overview (with 27 Appendices): A World Network Overview (with 27 Appendices):

Status and Plans for the Next Few Years of National & Status and Plans for the Next Few Years of National & Regional Networks, and Optical Network InitiativesRegional Networks, and Optical Network Initiatives

Monitoring Working Group ReportMonitoring Working Group Report [L. Cottrell] [L. Cottrell]

Digital Divide in RussiaDigital Divide in Russia [V. Ilyin] [V. Ilyin]

August 2004 Update Reports at the SCIC Web Site:August 2004 Update Reports at the SCIC Web Site:See http://icfa-scic.web.cern.ch/ICFA-SCIC/documents.htmSee http://icfa-scic.web.cern.ch/ICFA-SCIC/documents.htm

Asia Pacific, Latin America, GLORIAD (US-Ru-Asia Pacific, Latin America, GLORIAD (US-Ru-KoKo-China);-China);Brazil, Korea, etc. Brazil, Korea, etc.

SCIC Main Conclusion for 2003Setting the Tone for 2004

SCIC Main Conclusion for 2003Setting the Tone for 2004

The disparity among regions in HENP could increase The disparity among regions in HENP could increase even more sharply, as we learn to use advanced even more sharply, as we learn to use advanced networks effectively, and we develop dynamic Grid networks effectively, and we develop dynamic Grid systems in the most favored” regions systems in the most favored” regions

We must take action, and workWe must take action, and work to Close the Digital Divideto Close the Digital Divide To make Physicists from All World Regions Full To make Physicists from All World Regions Full

Partners in Their Experiments; and in the Process Partners in Their Experiments; and in the Process of Discoveryof Discovery

This is essential for the health of our global This is essential for the health of our global experimental collaborations, our plans for future experimental collaborations, our plans for future projects, and our field.projects, and our field.

ICFA Report: Networks for HENPICFA Report: Networks for HENPGeneral Conclusions (2)General Conclusions (2)

ICFA Report: Networks for HENPICFA Report: Networks for HENPGeneral Conclusions (2)General Conclusions (2)

Reliable high Reliable high End-to-end PerformanceEnd-to-end Performance of networked applications such as large of networked applications such as large file transfers and Data Grids is required. Achieving this requires:file transfers and Data Grids is required. Achieving this requires: End-to-end monitoring extending to all regions serving our community.End-to-end monitoring extending to all regions serving our community.

A coherent approach to monitoring that allows physicists throughout A coherent approach to monitoring that allows physicists throughout our community to extract clear information is required. our community to extract clear information is required.

Upgrading campus infrastructures.Upgrading campus infrastructures. These are still not designed to support Gbps data transfers in most HEP These are still not designed to support Gbps data transfers in most HEP centers. One reason for under-utilization of national and Int’l backbones, is centers. One reason for under-utilization of national and Int’l backbones, is the lack of bandwidth to end-user groups in the campus. the lack of bandwidth to end-user groups in the campus.

Removing local, last mile, and nat’l and int’l bottlenecks Removing local, last mile, and nat’l and int’l bottlenecks end-to-end, whether technical or political in origin. end-to-end, whether technical or political in origin.While National and International backbones have reached 2.5 to 10 Gbps While National and International backbones have reached 2.5 to 10 Gbps speeds in many countries, the bandwidths speeds in many countries, the bandwidths across borders, the countryside across borders, the countryside or the city may be much less. or the city may be much less.

This problem is very widespread in our community, This problem is very widespread in our community, with examples stretching from the Asia Pacific to with examples stretching from the Asia Pacific to Latin America Latin America to the Northeastern U.S. Root causes for this vary, to the Northeastern U.S. Root causes for this vary, from lack from lack of local infrastructure to unfavorable pricing policies. of local infrastructure to unfavorable pricing policies.

ICFA Report (2/2004) Update: ICFA Report (2/2004) Update: Main Trends Continue, Some AccelerateMain Trends Continue, Some Accelerate

ICFA Report (2/2004) Update: ICFA Report (2/2004) Update: Main Trends Continue, Some AccelerateMain Trends Continue, Some Accelerate

Current generation of 2.5-10 Gbps network backbones and major Int’l links Current generation of 2.5-10 Gbps network backbones and major Int’l links arrived in the last 2-3 Years [US+Europe+Japan; Now arrived in the last 2-3 Years [US+Europe+Japan; Now Korea and ChinaKorea and China]] Capability: 4 to Hundreds of Times; Much Faster than Moore’s LawCapability: 4 to Hundreds of Times; Much Faster than Moore’s Law

Proliferation of 10G links across the Atlantic Now; Will Begin use of Multiple Proliferation of 10G links across the Atlantic Now; Will Begin use of Multiple 10G Links (e.g. US-CERN) Along Major Paths by Fall 200510G Links (e.g. US-CERN) Along Major Paths by Fall 2005 Direct result of Falling Network Prices: $ 0.5 – 1M Per Year for 10GDirect result of Falling Network Prices: $ 0.5 – 1M Per Year for 10G

Ability to fully use long 10G paths with TCP continues to advance: Ability to fully use long 10G paths with TCP continues to advance: 7.5 Gbps X 16kkm (August 2004)7.5 Gbps X 16kkm (August 2004)

Technological progress driving equipment costs in end-systems lowerTechnological progress driving equipment costs in end-systems lower ““Commoditization” of Gbit Ethernet (GbE) ~complete: ($20-50 per port)Commoditization” of Gbit Ethernet (GbE) ~complete: ($20-50 per port)

10 GbE commoditization (e.g. < $ 2K per NIC with TOE) underway 10 GbE commoditization (e.g. < $ 2K per NIC with TOE) underway Some regions (US, Europe) moving to owned or leased dark fiber Some regions (US, Europe) moving to owned or leased dark fiber Emergence of the “Hybrid” Network Model:Emergence of the “Hybrid” Network Model: GNEW2004; UltraLight, GLIF GNEW2004; UltraLight, GLIF Grid-based AnalysisGrid-based Analysis demands end-to-end high performance & management demands end-to-end high performance & management The rapid rate of progress is confined mostly to the US, Europe, Japan and The rapid rate of progress is confined mostly to the US, Europe, Japan and

Korea, as well as the major Transatlantic routes; this Korea, as well as the major Transatlantic routes; this threatens to threatens to cause cause the Digital Divide to become a Chasmthe Digital Divide to become a Chasm

Work on the Digital Divide:Several Perspectives

Work on the Digital Divide:Several Perspectives

Work on Policies and/or Pricing: pk, in, br, cn, SE Europe, …Work on Policies and/or Pricing: pk, in, br, cn, SE Europe, … Find Ways to work with vendors, NRENs, and/or Gov’tsFind Ways to work with vendors, NRENs, and/or Gov’ts Exploit Model Cases: e.g. Poland, Slovakia, Czech RepublicExploit Model Cases: e.g. Poland, Slovakia, Czech Republic

Inter-Regional Projects Inter-Regional Projects GLORIAD,GLORIAD, Russia-China-US Optical Ring Russia-China-US Optical Ring South America: CHEPREO (US-Brazil); EU CLARA ProjectSouth America: CHEPREO (US-Brazil); EU CLARA Project Virtual SILK Highway Project (DESY): FSU satellite linksVirtual SILK Highway Project (DESY): FSU satellite links

Workshops and Tutorials/Training SessionsWorkshops and Tutorials/Training Sessions For Example: Digital Divide and HEPGrid Workshop,For Example: Digital Divide and HEPGrid Workshop,

UERJ Rio, February 2004; Next Daegu May 2005UERJ Rio, February 2004; Next Daegu May 2005 Help with Modernizing the Infrastructure Help with Modernizing the Infrastructure

Design, Commissioning, DevelopmentDesign, Commissioning, Development Tools for Effective Use: Monitoring, CollaborationTools for Effective Use: Monitoring, Collaboration

Participate in Standards Development; Open ToolsParticipate in Standards Development; Open Tools Advanced TCP stacks; Grid systemsAdvanced TCP stacks; Grid systems

Grid and Network Workshopat CERN March 15-16, 2004

Grid and Network Workshopat CERN March 15-16, 2004

WORKSHOP GOALS Share and challenge the lessons learned by nat’l and

international projects in the past three years; Share the current state of network engineering and

infrastructure and its likely evolution in the near future; Examine our understanding of the networking needs of

Grid applications (e.g., see the ICFA-SCIC reports); Develop a vision of how network engineering and

infrastructure will (or should) support Grid computing needs in the next three years.

CONCLUDING STATEMENT

"Following the 1st International Grid Networking Workshop(GNEW2004) that was held at CERN and co-organized byCERN/DataTAG, DANTE, ESnet, Internet2 & TERENA, there is a wide consensus that hybrid network services capable ofoffering both packet- and circuit/lambda-switching as well as highly advanced performance measurements and a newgeneration of distributed system software, will be required inorder to support emerging data intensive Grid applications, Such as High Energy Physics, Astrophysics, Climate andSupernova modeling, Genomics and Proteomics, requiring 10-100 Gbps and up over wide areas."

National Lambda Rail (NLR)National Lambda Rail (NLR)

15808 Terminal, Regen or OADM siteFiber route

NLRComing Up NowInitially 4 10G

WavelengthsNorthern Route

Operation by 4Q04 Internet2 HOPI Initiative (w/HEP) To 40 10G Waves in Future

Transition beginning now to optical, multi-wavelength Community owned or leased

“dark fiber” networks for R&E

PITPIT

PORPOR

FREFRE

RALRAL

WALWAL

NASNASPHOPHO

OLGOLGATLATL

CHICHI

CLECLE

KANKAN

OGDOGD

SACSAC BOSBOSNYCNYC

WDCWDC

STRSTR

DALDAL

DENDEN

LAXLAX

SVLSVL

SEASEA

SDGSDG

JACJAC nl, de, pl, cz,jp18 US States

JGN2: Japan Gigabit Network (4/04 – 3/08)20 Gbps Backbone, 6 Optical Cross-Connects

Kanazawa Sendai

Sapporo

Nagano

Kochi Nagoya

Fukuoka

Okinawa

Okayama

< １ G>　・ Teleport Okayama (Okayama) 　・ Hiroshima University (Higashi

Hiroshima) < １００ M>　・ Tottori University of

Environmental Studies (Tottori) 　・ Techno Ark Shimane (Matsue) 　・ New Media Plaza Yamaguchi

(Yamaguchi)

< １０ G>　・ Kyoto University (Kyoto) 　・ Osaka University (Ibaraki) < １ G>　 ・ NICT Kansai Advanced Research Center (Kobe) < １００ M>　・ Lake Biwa Data Highway AP * (Ohtsu)　・ Nara Prefectural Institute of Industrial

Technology (Nara) 　・ Wakayama University (Wakayama) 　・ Hyogo Prefecture Nishiharima Technopolis

(Kamigori-cho, Hyogo Prefecture) < １０ G> 　 ・ Kyushu University (Fukuoka) < １００ M>　・ NetCom Saga (Saga) 　・ Nagasaki University

(Nagasaki) 　・ Kumamoto Prefectural Office

(Kumamoto) 　・ Toyonokuni Hyper Network

AP *(Oita)　・ Miyazaki University (Miyazaki) 　・ Kagoshima University

(Kagoshima)

< １００ M>　・ Kagawa Prefecture Industry Promotion Center (Takamatsu) 　・ Tokushima University (Tokushima) 　・ Ehime University (Matsuyama) 　・ Kochi University of Technology

(Tosayamada-cho, Kochi Prefecture)

< １００ M>　・ Nagoya University (Nagoya) 　・ University of Shizuoka (Shizuoka) 　・ Softopia Japan (Ogaki, Gifu Prefecture) 　・ Mie Prefectural College of Nursing (Tsu)

< １０ G>　・ Ishikawa Hi-tech Exchange Center

(Tatsunokuchi-machi, Ishikawa Prefecture) < １００ M>　・ Toyama Institute of Information Systems (Toyama) 　・ Fukui Prefecture Data Super Highway AP * (Fukui) 　

< １００ M>　・ Niigata University

(Niigata) 　・ Matsumoto Information Creation Center

(Matsumoto, Nagano Prefecture)

< １０ G>　・ Tokyo University

(Bunkyo Ward, Tokyo) 　・ NICT Kashima Space Research Center

(Kashima, Ibaraki Prefecture) <1G>　・ Yokosuka Telecom Research Park

(Yokosuka, Kanagawa Prefecture) <100M>　・ Utsunomiya University (Utsunomiya) 　・ Gunma Industrial Technology Center

(Maebashi) 　・ Reitaku University

(Kashiwa, Chiba Prefecture) 　・ NICT Honjo Information and Communications Open Laboratory

(Honjo, Saitama Prefecture) 　・ Yamanashi Prefecture Open R&D Center

(Nakakoma-gun, Yamanashi Prefecture)

< １ G>　・ Tohoku University (Sendai) 　・ NICT Iwate IT Open Laboratory

(Takizawa-mura, Iwate Prefecture) < １００ M>　・ Hachinohe Institute of Technology

(Hachinohe, Aomori Prefecture) 　・ Akita Regional IX * (Akita) 　・ Keio University Tsuruoka Campus

(Tsuruoka, Yamagata Prefecture) 　・ Aizu University (Aizu Wakamatsu)

< １００ M>　 ・ Hokkaido Regional Network Association AP * (Sapporo)　　　　　　　　　　　　　　　　

NICT Tsukuba Research Center

20Gbps10Gbps1GbpsOptical testbeds

Core network nodes

Access points

Osaka

Otemachi

USA

NICT Keihannna Human Info-Communications

Research Center

NICT Kita Kyushu IT Open Laboratory

NICT Koganei Headquarters

[Legends ]

*IX:Internet eXchange

AP:Access Point

APAN-KR : KREONET/KREONet2 IIAPAN-KR : KREONET/KREONet2 II

Integrated hybrid experimental network, leveraging Transatlantic Integrated hybrid experimental network, leveraging Transatlantic R&D network partnerships; packet-switched + dynamic optical paths R&D network partnerships; packet-switched + dynamic optical paths 10 GbE across US and the Atlantic: NLR, DataTAG, TransLight, 10 GbE across US and the Atlantic: NLR, DataTAG, TransLight,

NetherLight, UKLight, etc.; Extensions to Japan, Taiwan, BrazilNetherLight, UKLight, etc.; Extensions to Japan, Taiwan, Brazil End-to-end monitoring; Realtime tracking and optimization; End-to-end monitoring; Realtime tracking and optimization;

Dynamic bandwidth provisioningDynamic bandwidth provisioning Agent-based services spanning all layers of the system,Agent-based services spanning all layers of the system, from the from the

optical cross-connects to the applications.optical cross-connects to the applications.

UltraLight Collaboration:UltraLight Collaboration:http://ultralight.caltech.eduhttp://ultralight.caltech.edu

Caltech, UF, FIU, UMich, SLAC,FNAL,MIT/Haystack,CERN, UERJ(Rio), NLR, CENIC, UCAID,Translight, UKLight, Netherlight, UvA, UCLondon, KEK, Taiwan

Cisco, Level(3)

GLIF: Global Lambda Integrated Facility

DWDM SURFnet

lambda service path

IP service path

10 Gbit/s

SURFnet10 Gbit/s

SURFnet10 Gbit/s

IEEAF10 Gbit/s

DwingelooASTRON/JIVE

DwingelooASTRON/JIVE

PragueCzechLight

PragueCzechLight

2.5 Gbit/s

NSF10 Gbit/s

LondonUKLightLondonUKLight

StockholmNorthernLightStockholm

NorthernLight

2.5 Gbit/s

New YorkMANLANNew YorkMANLAN

TokyoWIDETokyoWIDE

10 Gbit/s

10Gbit/s

10 Gbit/s

2x10 Gbit/s

IEEAF10 Gbit/s

2x10 Gbit/s

10 Gbit/s

2.5 Gbit/s

2.5 Gbit/s

TokyoAPANTokyoAPAN

GenevaCERN

GenevaCERN

ChicagoChicago AmsterdamAmsterdam

“ GLIF is a World Scale Lambda based Lab for Application and Middleware development, where Grid applications ride on dynamically configured networks based on optical wavelengths ... Coexisting with more traditional packet-switched network traffic

4th GLIF Workshop:Nottingham UK Sept. 2004

10 Gbps Wavelengths For R&E Network Development Are Prolifering,

Across Continents and Oceans

1660 km of Dark Fiber CWDM Links, up to 112 km.

1 to 4 Gbps (GbE)

August 2002: First NREN in Europe to establish Int’l GbE Dark Fiber Link, to Austria

April 2003 to Czech Republic.

Planning 10 Gbps Backbone; dark fiber link to Poland this year.

PROGRESS in SE Europe (Sk, Pl, Cz, Hu, …)

Dark Fiber in Eastern Europe Poland: PIONIER Network

Dark Fiber in Eastern Europe Poland: PIONIER Network

ŁÓDŹ

TORUŃ

P OZNAŃ

B YDGOS ZCZ

OLS ZTYN

B IAŁYS TOK

GDAŃS K

KOS ZALIN

S ZCZECIN

ZIELONAGÓRA

WROCŁAW

CZĘS TOCHOWA

KRAKÓW RZES ZÓW

LUB LIN

KIELCE

P UŁAWY

RADOM

KATOWICEGLIWICE

B IELS KO-B IAŁA

OP OLE

GUB IN WARS ZAWA

CIES ZYN

S IEDLCE

P IO NIE R n o d e sIn s ta lle d fib e r

F ib e rs p la n n e d in 2 0 0 4P IO NIE R n o d e s p la n n e d in 2 0 0 4

2650 2650 km Fiber km Fiber ConnectingConnecting

16 MANs; 5200 km 16 MANs; 5200 km and 21 MANs by 2005 and 21 MANs by 2005

Support Support Computational GridsComputational Grids

Domain-Specific Domain-Specific Grids Grids

Digital LibrariesDigital Libraries Interactive TVInteractive TV

Add’l Fibers for Add’l Fibers for e-Regional Initiatives e-Regional Initiatives

The Advantage of Dark Fiber CESNET Case Study (Czech Republic)

The Advantage of Dark Fiber CESNET Case Study (Czech Republic)

Case Study ResultCase Study ResultWavelength ServiceWavelength Service

Vs. Fiber Lease:Vs. Fiber Lease:

Cost Savings of Cost Savings of 50-70% Over 4 Years50-70% Over 4 Years

for Long 2.5G for Long 2.5G or 10G Links or 10G Links

2513 2513 km km Leased Fibers Leased Fibers (Since 1999) (Since 1999)

1 x 2,5G Leased 1 x 2,5G

(EURO/Month)about 150km (e.g. Ústí n.L. - Liberec) 7,000

about 300km (e.g. Praha - Brno) 8,000

***

4 x 2,5G Leased 4 x 2,5G

(EURO/Month)about 150km (e.g. Ústí n.L. - Liberec) 14,000

about 300km (e.g. Praha - Brno) 23,000

***

1 x 10G Leased 1 x 10G

(EURO/Month)about 150km (e.g. Ústí n.L. - Liberec) 14,000

about 300km (e.g. Praha - Brno) 16,000

***

4 x 10G Leased 4 x 10G

(EURO/Month)about 150km (e.g. Ústí n.L. - Liberec) 29,000

about 300km (e.g. Praha - Brno) 47,000

***

2 x booster 24dBm, 2 x DCF, DWDM 10G2 x (booster +In-line + preamplifier), 6 x DCF, DWDM 10G

Leased fibre with own equipment (EURO/Month)

5 000 *7 000 **

2 x booster 18dBm 2 x booster 27dBm + 2 x preamplifier + 6 x DCF

Leased fibre with own equipment (EURO/Month)

12 000 *14 000 **

8 000 **

2 x booster 21dBm, 2 x DCF2 x (booster 21dBm + in-line + preamplifier) + 6 x DCF

2 x booster 24dBm, DWDM 2,5G2 x (booster +In-line + preamplifier), 6 x DCF, DWDM 2,5G

Leased fibre with own equipment (EURO/Month)

5 000 *

Leased fibre with own equipment (EURO/Month)

8 000 *11 000 **

ICFA/SCIC Network Monitoring

Prepared by Les Cottrell, SLAC, forICFA

www.slac.stanford.edu/grp/scs/net/talk03/icfa-aug04.ppt

PingER: World View from SLAC

Important for policy makers

TCP throughput measured from N. America to World Regions

1

10

100

1000

10000J

an

-95

Ja

n-9

6

De

c-9

6

De

c-9

7

De

c-9

8

De

c-9

9

De

c-0

0

De

c-0

1

De

c-0

2

De

c-0

3

De

c-0

4

Deri

ved

TC

P t

hro

ug

hp

ut

in K

Byte

s/s

ec

1

10

100

1000

10000

China (13)

S.E. Europe (21)

Europe(150) Canada (27)

Russia(17)

Edu (141)

Latin America (37)

India(7) Africa (30)

Mid East (16)

50% Improvement/year~ factor of 10 in < 6 years

C. Asia (8)

From the PingER project, Aug 2004

Caucasus (8)

C. Asia, Russia, SE Europe, L. America, M. East, China:

4-5 yrs behindIndia, Africa: 7 yrs behind

View from CERNConfirms This View

Now monitoring 650 sites in 115 countriesIn last 9 months:

Several sites in Russia (GLORIAD)Many hosts in Africa (27 of 54 Countries) Monitoring sites in Pakistan, Brazil

Research Networking in Latin America: August 2004

The only Countries with research The only Countries with research network connectivity now in network connectivity now in Latin America:Latin America:Argentina, Brazil, Chile, Argentina, Brazil, Chile, Mexico, Venezuela Mexico, Venezuela

AmPathAmPath Provided connectivity for Provided connectivity for some South American countriessome South American countries

New Sao Paolo-Miami Link at 622 MbpsNew Sao Paolo-Miami Link at 622 MbpsStarting This MonthStarting This Month

New: New: CLARA CLARA (Funded by EU)(Funded by EU) Regional Network Connecting 19 Countries:Regional Network Connecting 19 Countries:

AmPath

Argentina Argentina Dominican Republic Dominican Republic Panama Panama Brasil Brasil Ecuador Ecuador ParaguayParaguayBolivia Bolivia El Salvador El Salvador PeruPeruChile Chile Guatemala Guatemala UruguayUruguayColombia Colombia Honduras Honduras VenezuelaVenezuelaCosta Rica Mexico Costa Rica Mexico NicaraguaNicaraguaCubaCuba

155 Mbps Backbone with 10-45 Mbps Spurs;

4 Mbps Satellite to Cuba; 622 Mbps to Europe

Also NSF Proposals To

Connect at 2.5G to US

HEPGRID (CMS) in BrazilHEPGRID-CMS/BRAZIL is a project to build a Grid thatAt Regional Level will include CBPF,UFRJ,UFRGS,UFBA, UERJ & UNESPAt International Level will be integrated with CMS Grid based at CERN; focal points include iVGDL/Grid3 and bilateral projects with Caltech Group

France Italy USAGermany BRAZIL622 Mbps

UFRGS

UERJUFRJ

T1

IndividualMachines

On line systems

Brazilian HEPGRID

CBPF

UNESP/USPSPRACE-Working

Gigabit

CERN 2.5 - 10 Gbps

UFBA

UERJ RegionalTier2 Ctr

T4

T0 +T1

T2 T1

T3 T2

UERJ: T2T1,

100500 Nodes;

Plus T2s to

100 Nodes

Subject Argentina Brazil Chile Colombia Costa Rica

Equator Mexico

Astrophysics

e-VLBI

High Energy Physics

Geosciences

Marine sciences

Health and Biomedical applications

Environmental studies

Latin America Science Areas Interested in Improving Connectivity ( by Country)

Networks and Grids: The Potential to Spark a New Era of Science in the Region

Asia Pacific Academic Network ConnectivityAsia Pacific Academic Network Connectivity

Access PointExchange PointCurrent Status2004 (plan)

US

Europe

ID

VN

LK

KR

TH

MY

SG

RU

PH

CN

HK

JP2G

12M

34M

45M

20.9G

155M

777M

1.2G

TW

200M

310M

AU

45M`722M

IN

16M

1.5M

1.5M

7.5M2M

2.5M

2M

90M

155M

622M

155M

9.1G

932M(to 21 G)

Better Better North/South Linkages within AsiaNorth/South Linkages within AsiaJP-SG link: 155Mbps in 2005 is proposed to NSF by JP-SG link: 155Mbps in 2005 is proposed to NSF by CIRENCIRENJP- TH link: 2Mbps JP- TH link: 2Mbps 45Mbps in 2004 is being 45Mbps in 2004 is being studied.studied.CIREN is studying an extension to IndiaCIREN is studying an extension to India

Connectivity Connectivity to US from JP, to US from JP, KO, AU is KO, AU is Advancing Advancing Rapidly.Rapidly.

Progress in Progress in the Region, the Region, and to Europe and to Europe is Much is Much Slower Slower

APAN Status July APAN Status July 20042004

APAN Link Information 2004.7.7 sec@apan.net

Countries Network Bandwidth (Mbps) AUP/RemarkAU-US AARNet 310 to 2 x 10 Gbps soon R&E + CommodityAU-US (PAIX) AARNet 622 R&E + CommodityCN-HK CERNET 622 R&E + CommodityCN-HK CSTNET 155 R&ECN-JP CERNET 155 R&ECN-JP CERNET 45 Native IPv6CN-US CERNET 155 R&E + CommodityCN-US CSTNET 155 R&EHK-US HARNET 45 R&EHK-TW HARNET/TANET/ASNET 100 R&EIN-US/UK ERNET 16 R&EJP-ASIA UDL 9 R&EJP-ID AI3(ITB) 0.5/1.5 R&EJP-KR APII 2Gbps R&EJP-LA AI3 (NUOL) 0.128/0.128 R&EJP-MY AI3(USM) 1.5/0.5 R&EJP-PH AI3(ASTI) 1.5/0.5 R&EJP-PH MAFFIN 6 Research JP-SG AI3(TP) 1.5/0.5 R&EJP-TH AI3(AIT) (service interrupted) R&EJP-TH SINET(ThaiSarn) 2 R&EJP-US TransPac 5 Gbps to 2x10 Gbps soon R&E

APAN Link Information 2004.7.7 sec@apan.net

Countries Network Bandwidth (Mbps) AUP/Remark(JP)-US-EU SINET 155 R&E / No TransitJP-US SINET 5 Gbps R&E / No TransitJP-US IEEAF 10Gbps R&E wave serviceJP-US IEEAF 622 R&EJP-US Japan-Hawaii 155 R&EJP-VN AI3(IOIT) 1.5/0.5 R&EKR-FR KOREN/RENATER 34 Research (TEIN) KR-SG APII 8 R&EKR-US KOREN/KREONet2 1.2Gbps R&ELK-JP LEARN 2.5 R&E MY-SG NRG/SICU 2 Experiment (Down)SG-US SingaREN 90 R&ETH-US Uninet 155 R&E TW-HK ASNET/TANET/TWAREN 622 R&ETW-JP ASNET/TANET 622 R&ETW-SG ASNET/SingAREN 155 R&ETW-US ASNET/TANET/TWAREN 6.6 Gbps R&E(TW)-US-NL ASNET/TANET/TWAREN 2.5 Gbps R&E

APAN Recommendations(at July 2004 Meeting in CAIRNS, Au)

APAN Recommendations(at July 2004 Meeting in CAIRNS, Au)

Central Issues for APAN this decadeCentral Issues for APAN this decade Stronger linkages between applications and infrastructure - Stronger linkages between applications and infrastructure -

neither can exist independentlyneither can exist independently Stronger application and infrastructure linkages among APAN Stronger application and infrastructure linkages among APAN

members. members. Continuing focus on APAN as an organization that represents Continuing focus on APAN as an organization that represents

infrastructure interests in Asiainfrastructure interests in Asia Closer connection between APAN the infrastructure & Closer connection between APAN the infrastructure &

applications organization and regional political organizations applications organization and regional political organizations (e.g. APEC, ASEAN)(e.g. APEC, ASEAN)

New issues demand attentionNew issues demand attention Application measurement, particularly end-to-end network Application measurement, particularly end-to-end network

performance measurement is increasingly critical (deterministic performance measurement is increasingly critical (deterministic networking)networking)

Security must now be a consideration for every application and Security must now be a consideration for every application and every network.every network.

KR-US/CA Transpacific connectionKR-US/CA Transpacific connection

Participation in Global-scale Lambda NetworkingParticipation in Global-scale Lambda Networking Two STM-4 circuits (1.2G) : KR-CA-USTwo STM-4 circuits (1.2G) : KR-CA-US Global lambda networking : North America, Europe, Global lambda networking : North America, Europe,

Asia Pacific, etc.Asia Pacific, etc.

STM-4 * 2

Seattle

APII-testbed/KREONet2

KREONET/SuperSIReNCA*Net4

Global Lambda Networking

StarLight

Chicago

PacWave

New Record!!! 916 Mbps from CHEP to Caltech New Record!!! 916 Mbps from CHEP to Caltech (22/06/’04)(22/06/’04)

Subject: UDP test on KOREN-TransPAC-CaltechSubject: UDP test on KOREN-TransPAC-Caltech

Date: Tue, 22 Jun 2004 13:47:25 +0900Date: Tue, 22 Jun 2004 13:47:25 +0900

From: "Kihwan Kwon" <kihwan@bh.knu.ac.kr>From: "Kihwan Kwon" <kihwan@bh.knu.ac.kr>

To: <son@knu.ac.kr>To: <son@knu.ac.kr>

[root@sul Iperf]# ./iperf -c socrates.cacr.caltech.edu -u -b 1000m [root@sul Iperf]# ./iperf -c socrates.cacr.caltech.edu -u -b 1000m

------------------------------------------------------------------------------------------------------------------------

Client connecting to socrates.cacr.caltech.edu, UDP port 5001Client connecting to socrates.cacr.caltech.edu, UDP port 5001

Sending 1470 byte datagrams; UDP buffer size: 64.0 KByte (default)Sending 1470 byte datagrams; UDP buffer size: 64.0 KByte (default)

------------------------------------------------------------------------------------------------------------------------

[ 5] local 155.230.20.20 port 33036 connected with 131.215.144.227[ 5] local 155.230.20.20 port 33036 connected with 131.215.144.227

[ ID] Interval Transfer Bandwidth[ ID] Interval Transfer Bandwidth

[ 5] 0.0-2595.2 sec 277 GBytes 916 Mbits/sec[ 5] 0.0-2595.2 sec 277 GBytes 916 Mbits/sec

USA TransPACUSA TransPAC

G/H-JapanG/H-JapanKNU/KoreaKNU/Korea

Max. 947.3Mbps

OC3 circuits Moscow-Chicago-OC3 circuits Moscow-Chicago-Beijing since January 2004Beijing since January 2004

OC3 circuit Moscow-Beijing July OC3 circuit Moscow-Beijing July 2004 (completes the ring)2004 (completes the ring)

Korea (KISTI) joining US, Russia, Korea (KISTI) joining US, Russia, China as full partner in GLORIADChina as full partner in GLORIAD

Plans developing for Central Asian Plans developing for Central Asian extension (w/Kyrgyz Government)extension (w/Kyrgyz Government)

Rapid traffic growth with heaviest Rapid traffic growth with heaviest US use from DOE (FermiLab), US use from DOE (FermiLab), NASA, NOAA, NIH and Universities NASA, NOAA, NIH and Universities (UMD, IU, UCB, UNC, UMN, PSU, (UMD, IU, UCB, UNC, UMN, PSU, Harvard, Stanford, Wash., Oregon, Harvard, Stanford, Wash., Oregon, 250+ others) 250+ others)

GLORIAD 5-year Proposal Pending (with US NSF) for expansion: 2.5G Moscow-Amsterdam-Chicago-Seattle-Hong Kong-Pusan-Beijing circuits early 2005; 10G ring around northern hemisphere 2007; multiple wavelength service 2009 – providing hybrid circuit-switched (primarily Ethernet) and routed services

> 5TBytes now transferred monthly via GLORIAD to US, Russia, China

Global Ring Network for Advanced Applications Development

Aug. 8 2004: P.K. Young, Korean IST

Advisor to President Announces

Korea Joining GLORIAD TEIN gradually to 10G, connected to GLORIAD Asia Pacific Info. Infra- Structure (1G) will be

backup net to GLORIAD

Internet in China (J.P.Wu APAN July 2004)

Internet users in China: from 6.8 Million to 78 Million within 6 months

IP Addresses: 32M （ 1A+233B+146C ）Backbone ： 2.5-10G DWDM+RouterInternational links ： 20GExchange Points ： > 30G （ BJ ， SH ， GZ ）Last Miles Ethernet ， WLAN ， ADSL ， CTV ， CDMA ， ISDN

， GPRS ， Dial-upNeed IPv6

TotalWirelin

eDial Up ISDN

Broadband

68M 23.4M 45.0M 4.9M 9.8M

 

China: CERNET Update1995, 64K Nation wide backbone connecting 8 cities, 100 Universities1998, 2M Nation wide backbone connecting 20 cities, 300 Universities2000, Own dark fiber crossing 30+ major cities and 30,000 kilometers2001, CERNET DWDM/SDH network finished2001, 2.5G/155M Backbone connecting 36 cities, 800 universities2003,1300 + universities and institutes, over 15 million users

CERNET2 and Key TechnologiesCERNET 2: Next Generation Education and Research Network in ChinaCERNET 2 Backbone connecting 15-20 GigaPOPs at 2.5G-10Gbps (I2-like Model)Connecting 200 Universities and 100+ Research Institutes at 1Gbps-10GbpsNative IPv6 and Lambda Networking Support/Deployment of the following technologies: E2E performance monitoring Middleware and Advanced Applications Multicast

AFRICA: Key TrendsAFRICA: Key Trends

Growth in traffic and lack of infrastructure Growth in traffic and lack of infrastructure Predominance of Satellite;Predominance of Satellite;But these satellites are heavily subscribed But these satellites are heavily subscribed

Int’l Links: Only ~1% of traffic on links is for Internet connections;Int’l Links: Only ~1% of traffic on links is for Internet connections;Most Internet traffic (for ~80% of countries) via satelliteMost Internet traffic (for ~80% of countries) via satellite Flourishing Grey market for Internet & VOIP traffic using VSAT dishesFlourishing Grey market for Internet & VOIP traffic using VSAT dishes

Many Regional fiber projects in “planning phase” (some languished in Many Regional fiber projects in “planning phase” (some languished in the past); Only links from South Africa to Nimibia, Botswana done so far the past); Only links from South Africa to Nimibia, Botswana done so far

Int’l fiber Project: SAT-3/WASC/SAFE Cable from South Africa to PortugalInt’l fiber Project: SAT-3/WASC/SAFE Cable from South Africa to PortugalAlong West Coast of AfricaAlong West Coast of Africa Supplied by Alcatel to Worldwide Consortium of 35 CarriersSupplied by Alcatel to Worldwide Consortium of 35 Carriers 40 Gbps by Mid-2003; Heavily Subscribed. Ultimate Capacity 120 Gbps40 Gbps by Mid-2003; Heavily Subscribed. Ultimate Capacity 120 Gbps Extension to Interior Mostly by Satellite: < 1 Mbps to ~100 Mbps typical Extension to Interior Mostly by Satellite: < 1 Mbps to ~100 Mbps typical

M. Jensen and P. Hamilton Infrastructure Report, March 2004

Note: World Conference on Physics and Sustainable Development,

10/31 – 11/2/05 in Durban South Africa; Part of World Year of Physics 2005.

Sponsors: UNESCO, ICTP, IUPAP, APS, SAIP

AFRICA:AFRICA: Nectar Net Initiative Nectar Net Initiative

Growing Need to connect academic researchers, medicalGrowing Need to connect academic researchers, medicalresearchers & practitioners to many sites in Africaresearchers & practitioners to many sites in Africa

ExamplesExamples: : CDC & NIH: Global AIDS Project, Dept. of Parasitic Diseases,CDC & NIH: Global AIDS Project, Dept. of Parasitic Diseases,

Nat’l Library of Medicine (Ghana, Nigeria)Nat’l Library of Medicine (Ghana, Nigeria) Gates $ 50M HIV/AIDS Center in BotswanaGates $ 50M HIV/AIDS Center in Botswana; Project Coord at Harvard; Project Coord at Harvard Africa Monsoon AMMA Project, Dakar Site [cf. East US Hurricanes]Africa Monsoon AMMA Project, Dakar Site [cf. East US Hurricanes] US Geological Survey: Global Spatial Data InfrastructureUS Geological Survey: Global Spatial Data Infrastructure Distance Learning: Emory-Ibadan (Nigeria); Research Distance Learning: Emory-Ibadan (Nigeria); Research

Channel ContentChannel ContentBut Africa is HardBut Africa is Hard: : 11M Sq. Miles, 600 M People, 54 Countries11M Sq. Miles, 600 M People, 54 Countries

Little Little Telecommunications InfrastructureTelecommunications Infrastructure

Approach:Approach: Use SAT-3/WASC Cable (to Portugal), GEANT Use SAT-3/WASC Cable (to Portugal), GEANT Across Europe, Amsterdam-NY Link Across the Atlantic, Across Europe, Amsterdam-NY Link Across the Atlantic, then Peer with R&E Networks such as Abilene in NYCthen Peer with R&E Networks such as Abilene in NYC Cable Landings in 8 West African Countries and South AfricaCable Landings in 8 West African Countries and South Africa Pragmatic approach to reach end points: Pragmatic approach to reach end points:

VSAT satellite, ADSL, microwave, etc.VSAT satellite, ADSL, microwave, etc.W. MatthewsGeorgia Tech

Sample Bandwidth Costs for African Universities

$0.27

$3.00

$6.77

$9.84

$11.03

$20.00

$0.00 $5.00 $10.00 $15.00 $20.00 $25.00

USA

IBAUD Target

Ghana

Uganda

Average

Nigeria

$/kbps/month

Sample size of 26 universitiesAverage Cost for VSAT service: Quality, CIR, Rx, Tx not distinguished

Bandwidth prices in Africa vary dramatically; are in general many times what they could be if universities purchase in volume

Roy Steiner Internet2 Workshop

27 sites (U.S., Korea) 2300-2800 CPUs 700-1100 Concurrent Jobs

Korea

www.ivdgl.org/grid2003

Grid2003: An Operational Grid2003: An Operational Production Grid, Since October 2003Production Grid, Since October 2003

Trillium: PPDGGriPhyNiVDGL

Prelude to Open Science Grid: www.opensciencegrid.org

The original Computational and Data Grid concepts are The original Computational and Data Grid concepts are largely stateless, open systems largely stateless, open systems

Analogous to the WebAnalogous to the WebThe classical Grid architecture had a number of implicit The classical Grid architecture had a number of implicit

assumptionsassumptions The ability to locate and schedule suitable resources,The ability to locate and schedule suitable resources,

within a tolerably short time (i.e. resource richness) within a tolerably short time (i.e. resource richness) Short transactions with relatively simple failure modesShort transactions with relatively simple failure modes

HENP Grids are HENP Grids are Data Intensive & Resource-ConstrainedData Intensive & Resource-Constrained Resource usage governed by local and global policiesResource usage governed by local and global policies Long transactions; some long queuesLong transactions; some long queues Analysis:Analysis: 1000s of users competing for resources 1000s of users competing for resources at dozens of sites: complex scheduling; management at dozens of sites: complex scheduling; management

HENP HENP Stateful, End-to-end Monitored and Tracked ParadigmStateful, End-to-end Monitored and Tracked Paradigm Adopted in OGSA [Now WS Resource Framework]Adopted in OGSA [Now WS Resource Framework]

HENP Data Grids, and Now HENP Data Grids, and Now Services-Oriented GridsServices-Oriented Grids

The Move to OGSA and then The Move to OGSA and then Managed Integrated SystemsManaged Integrated Systems

Incr

ease

d f

un

ctio

nal

ity,

stan

dar

diz

atio

n

Time

Customsolutions

Open GridServices Arch

GGF: OGSI, …(+ OASIS, W3C)

Multiple implementations,including Globus Toolkit

Web services + …

Globus Toolkit

Defacto standardsGGF: GridFTP, GSI

X.509,LDAP,FTP, …

App-specificServices~~Integrated SystemsIntegrated Systems

Stateful; ManagedWeb Services

Resrc Framwk

Managing Global Systems: Dynamic Managing Global Systems: Dynamic Scalable Services ArchitectureScalable Services Architecture

MonALISA: http://monalisa.cacr.caltech.edu

““Station Server” Station Server” Services-enginesServices-engines at sites host many at sites host many“Dynamic Services”“Dynamic Services” Scales to Scales to

thousands of thousands of service-Instancesservice-Instances

Servers autodiscover Servers autodiscover and interconnect and interconnect dynamically to form dynamically to form a robust fabric a robust fabric

Autonomous agentsAutonomous agents

+ CLARENS: Web Services Fabric and Portal Architecture

24 24 X 7 OperationsX 7 OperationsMultiple Orgs.Multiple Orgs.

Grid2003Grid2003 US CMSUS CMS CMS-DC04CMS-DC04 ALICEALICE STARSTAR VRVSVRVS ABILENEABILENE Soon: GEANTSoon: GEANT + GLORIAD+ GLORIAD

52Caltech GAE Team

Grid Analysis EnvironmentGrid Analysis Environment

Analysis Clients talk standard protocols to the CLARENS “Grid Services Web Server”, with a simple Web service API

The secure Clarens portal hides the complexity of the Grid Services from the client

Key features: Global Scheduler, Catalogs, Monitoring, and Grid-wide Execution service;Clarens servers forma Global Peer to peer Network

SchedulerCatalogs

Analysis Client

Grid ServicesWeb Server

ExecutionPriority

Manager

Grid WideExecutionService

DataManagement

Fully-ConcretePlanner

Fully-AbstractPlanner

Analysis Client

AnalysisClient

Virtual Data

Replica

ApplicationsMonitoring

Partially-AbstractPlanner

Metadata

HTTP, SOAP, XML/RPC

CLARENS: Web Services Architecture

World Summit on the Information Society(WSIS): Geneva 12/2003 and Tunis in 2005 World Summit on the Information Society

(WSIS): Geneva 12/2003 and Tunis in 2005 The UN General Assembly adopted in 2001 a resolution The UN General Assembly adopted in 2001 a resolution

endorsing the organization of the endorsing the organization of the World Summit on the World Summit on the Information Society (WSIS),Information Society (WSIS), under UN Secretary-General, under UN Secretary-General, Kofi Annan, with the ITU and host governments taking Kofi Annan, with the ITU and host governments taking the lead role in its preparation. the lead role in its preparation.

GOAL: To Create an Information Society:GOAL: To Create an Information Society: A Common Definition was adoptedA Common Definition was adoptedin the “Tokyo Declaration” of January 2003:in the “Tokyo Declaration” of January 2003:“… One in which highly developed ICT networks, equitable “… One in which highly developed ICT networks, equitable and ubiquitous access to information, appropriate content and ubiquitous access to information, appropriate content in accessible formats and effective communication can in accessible formats and effective communication can help people achieve their potential”help people achieve their potential”

Kofi Annan Challenged the Scientific Community to Help (3/03) Kofi Annan Challenged the Scientific Community to Help (3/03) CERN and ICFA SCIC have been quite active in the WSIS in CERN and ICFA SCIC have been quite active in the WSIS in

Geneva (12/2003) Geneva (12/2003)

Role of Science in the Information Society; WSIS 2003-2005

Role of Science in the Information Society; WSIS 2003-2005

HENP Active in WSIS HENP Active in WSIS CERN RSIS EventCERN RSIS EventSIS Forum & CERN/Caltech SIS Forum & CERN/Caltech

Online Stand at WSIS I Online Stand at WSIS I (Geneva 12/03) (Geneva 12/03)

Visitors at WSIS I Visitors at WSIS I Kofi AnnanKofi Annan, UN Sec’y General , UN Sec’y General John H. MarburgerJohn H. Marburger, ,

Science Adviser to US PresidentScience Adviser to US PresidentIon IliescuIon Iliescu, President of Romania; , President of Romania;

and Dan Nica, Minister of ICTand Dan Nica, Minister of ICTJean-Paul Hubert, Ambassador Jean-Paul Hubert, Ambassador

of Canada in Switzerland of Canada in Switzerland……

Planning Underway forPlanning Underway for WSIS II: Tunis 2005 WSIS II: Tunis 2005

NEWS: Bulletin: ONE TWOWELCOME BULLETIN General InformationRegistrationTravel Information Hotel RegistrationParticipant List How to Get UERJ/Hotel Computer AccountsUseful Phone Numbers ProgramContact us: Secretariat Chairmen

 CLAF  CNPQ FAPERJ        UERJ

SPONSORS

HEPGRID and Digital Divide Workshop

UERJ, Rio de Janeiro, Feb. 16-20 2004Theme: Global Collaborations, Grids

and Their Relationship to the Digital Divide

ICFA, understanding the vital role of these issues for our field’s future, commissioned the Standing Committee on Inter-regional Connectivity (SCIC) in 1998, to survey and

monitor the state of the networks used by our field, and identify problems. For the past

three years the SCIC has focused on understanding and seeking the means of reducing or eliminating the Digital Divide, and proposed to ICFA that these issues, as

they affect our field of High Energy Physics, be brought to our community for discussion. This led to ICFA’s approval, in July 2003, of the Digital Divide and HEP Grid Workshop. 

More Information: http://www.lishep.uerj.br

TutorialsC++Grid TechnologiesGrid-Enabled

AnalysisNetworksCollaborative Systems

Sessions &Tutorials Available

(w/Video) on the Web

Computing Model Progress CMS Internal Review of Software and Computing

International ICFA Workshop on HEP International ICFA Workshop on HEP Networking, Grids and Digital Divide Networking, Grids and Digital Divide Issues for Global e-ScienceIssues for Global e-Science

Proposed Workshop Dates: May 23-27, 2005

Venue: Daegu, Korea

Dongchul SonCenter for High Energy PhysicsKyungpook National University

ICFA, Beijing, ChinaAug. 2004 ICFA Approval

Requested Today

고에너지물리연구센터 CENTER FOR HIGH ENERGY PHYSICS

International ICFA Workshop on HEP International ICFA Workshop on HEP Networking, Grids Networking, Grids and Digital Divide Issues for Global e-Scienceand Digital Divide Issues for Global e-Science

ThemesNetworking, Grids, and Their Relationship to the Digital Divide for

HEP as Global e-ScienceFocus on Key Issues of Inter-regional Connectivity

Mission Statement

ICFA, understanding the vital role of these issues for our field’s future, commissioned the Standing Committee on Inter-regional Connectivity (SCIC) in 1998, to survey and monitor the state of the networks used by our field, and identify problems. For the past three years the SCIC has focused on understanding and seeking the means of reducing or eliminating the Digital Divide, and proposed to ICFA that these issues, as they affect our field of High Energy Physics, be brought to our community for discussion. This workshop, the second in the series begun with the the 2004 Digital Divide and HEP Grid Workshop in Rio de Janeiro (approved by ICFA in July 2003) will carry forward this work while strengthening the involvement of scientists, technologists and governments in the Asia Pacific region.

고에너지물리연구센터 CENTER FOR HIGH ENERGY PHYSICS

International ICFA Workshop on HEP International ICFA Workshop on HEP Networking, Grids Networking, Grids and Digital Divide Issues for Global e-Scienceand Digital Divide Issues for Global e-Science

Workshop Goals Review the current status, progress and barriers to the effective use

of the major national, continental and transoceanic networks used by HEP

Review progress, strengthen opportunities for collaboration, and explore the means to deal with key issues in Grid computing and Grid-enabled data analysis, for high energy physics and other fields of data intensive science, now and in the future

Exchange information and ideas, and formulate plans to develop solutions to specific problems related to the Digital Divide in various regions, with a focus on Asia Pacific, Latin America, Russia and Africa

Continue to advance a broad program of work on reducing or eliminating the Digital Divide, and ensuring global collaboration, as related to all of the above aspects.

Networks and Grids, GLORIAD, ITER and HENP

Networks and Grids, GLORIAD, ITER and HENP

Network backbones and major links used by major experimentsNetwork backbones and major links used by major experiments in HENP and other fields are advancing rapidly in HENP and other fields are advancing rapidly

To the 10 G range in < 2 years; much faster than Moore’s LawTo the 10 G range in < 2 years; much faster than Moore’s Law New HENP and DOE Roadmaps: a factor ~1000 improvement per decadeNew HENP and DOE Roadmaps: a factor ~1000 improvement per decade

We are learning to use long distance 10 Gbps networks effectively We are learning to use long distance 10 Gbps networks effectively 2003-2004 Developments: to 7.5 Gbps flows over 16 kkm2003-2004 Developments: to 7.5 Gbps flows over 16 kkm Important advances in Asia-Pacific, notably KoreaImportant advances in Asia-Pacific, notably Korea

A transition to community-owned and operated R&E networks A transition to community-owned and operated R&E networks is beginning (us, ca, nl, pl, cz, sk …) or considered (de, ro, …) is beginning (us, ca, nl, pl, cz, sk …) or considered (de, ro, …)

We Must Work to Close to Digital DivideWe Must Work to Close to Digital Divide Allowing Scientists and Students from All World Regions Allowing Scientists and Students from All World Regions

to Take Part in Discoveries at the Frontiers of Science to Take Part in Discoveries at the Frontiers of Science Removing Regional, Last Mile, Local Bottlenecks and Removing Regional, Last Mile, Local Bottlenecks and

Compromises in Network Quality are now On Compromises in Network Quality are now On the Critical Path the Critical Path GLORIADGLORIAD is A Key Project to Achieve These Goals is A Key Project to Achieve These Goals

Synergies Between the Data-Intensive Missions of HENP & ITERSynergies Between the Data-Intensive Missions of HENP & ITER Enhancing Partnership and Community among the US, Russia Enhancing Partnership and Community among the US, Russia

and China: both in Science and Education and China: both in Science and Education

SC2004: HEP Network LayoutPreview of Future Grid systems

SC2004: HEP Network LayoutPreview of Future Grid systems

Joint Caltech, CERN, SLAC, Joint Caltech, CERN, SLAC, FNAL, UKlight, HP, Cisco… DemoFNAL, UKlight, HP, Cisco… Demo

6 to 8 10 Gbps waves to HEP6 to 8 10 Gbps waves to HEP setup on the show floor setup on the show floor

Bandwidth challenge: aggregate Bandwidth challenge: aggregate throughput goal throughput goal of 40 to 60 Gbps of 40 to 60 Gbps

3*10GbpsTeraGrid

Australia

JapanBrazil

SLACUK

10 GbpsAbilene

10 GbpsNLR

2*10 GbpsNLR

Caltech CACR CERN

Geneva

StarLight

LA

10 GbpsLHCNet 2 Metro

10 Gbps Waves LA-Caltech

FNAL

SCIC in 2003-2004http://cern.ch/icfa-scic

SCIC in 2003-2004http://cern.ch/icfa-scic

Strong Focus on the Strong Focus on the Digital DivideDigital Divide Continues Continues A Striking Picture Continues to Emerge: Remarkable A Striking Picture Continues to Emerge: Remarkable

Progress in Some Regions, and a Deepening Digital Progress in Some Regions, and a Deepening Digital Divide Among NationsDivide Among Nations

IntensiveIntensive Work in the Field: > 60 Meetings and Workshops: Work in the Field: > 60 Meetings and Workshops: E.g., Internet2, TERENA, AMPATH, APAN, CHEP2003, SC2003, E.g., Internet2, TERENA, AMPATH, APAN, CHEP2003, SC2003,

Trieste, Telecom World 2003, SC2003, WSIS/RSIS, GLORIAD Trieste, Telecom World 2003, SC2003, WSIS/RSIS, GLORIAD LaunchLaunch, Digital Divide and HEPGrid Workshop (Feb. 16-20 in , Digital Divide and HEPGrid Workshop (Feb. 16-20 in Rio), Rio), GNEW2004, GridNets2004, NASA ONT Workshop, … etc. GNEW2004, GridNets2004, NASA ONT Workshop, … etc.

33rdrd Int’l Grid Workshop in Daegu (August 26-28, 2004); Plan for Int’l Grid Workshop in Daegu (August 26-28, 2004); Plan for 22ndnd ICFA Digital Divide and Grid Workshop in Daegu (May 2005) ICFA Digital Divide and Grid Workshop in Daegu (May 2005)

HENP increasingly visible to governments; heads of state:HENP increasingly visible to governments; heads of state: Through Network advances (records), Grid developments,Through Network advances (records), Grid developments,

Work on the Digital Divide and issues of Global Collaboration Work on the Digital Divide and issues of Global Collaboration Also through the World Summit on the Information Society Also through the World Summit on the Information Society

Process. Next Step is Process. Next Step is WSIS II in TUNIS November 2005WSIS II in TUNIS November 2005

Coverage Now monitoring 650 sites in 115 countries In last 9 months added:

Several sites in Russia (thanks GLORIAD)Many hosts in Africa (5 36 now; in 27 out of 54 countries)Monitoring sites in Pakistan and Brazil (Sao Paolo and Rio)

Working to install monitoring host in Bangalore, India

Monitoring siteRemote site

Latin America: CLARA Network (2004-2006 EU Project)

Significant contribution from European Comission and Dante through ALICE project

NRENs in 18 LA countries forming a regional network for collaboration traffic

Initial backbone ring bandwidth f 155 Mbps

Spur links at 10 to 45 Mbps (Cuba at 4 Mbps by satellite)

Initial connection to Europe at 622 Mbps from Brazil

Tijuana (Mexico) PoP soon to be connected to US through dark fibre link (CUDI-CENIC) access to US, Canada and Asia -

Pacific Rim

To East Coast

ClaraClara

To West Coast

to West Coast to East Coast

to Europ

eNSF IRNC 2004: Two Proposals to

Connect CLARA to the US (and Europe)1st Proposal:

FIU and CENIC2nd Proposal:

Indiana and Internet2

Note: CHEPREO (FIU, UF, FSU Caltech, UERJ, USP, RNP)

622 Mbps Sao Paolo – Miami Link Started in August

GIGA Project: Experimental Gbps Network: Sites in Rio and Sao Paolo

FapesptelcosUnesp

USP – IncorUSP - C.Univ.

CPqDLNLS

Unicamp

LNCC

CPTEC UFF

CTAINPE

CBPFLNCC

FiocruzIME

IMPA-RNP

PUC-Rio telcosUERJUFRJ

UniversitiesIMEPUC-RioUERJUFFUFRJUnespUnicampUSP

R&D CentresCBPF - physicsCPqD - telecomCPTEC - meteorologyCTA - aerospaceFiocruz - healthIMPA - mathematicsINPE - space sciencesLNCC - HPCLNLS - physics

About 600 km extension - not to scale

Slide from M. Stanton

Trans-Eurasia Information NetworkTEIN (2004-2007)

Trans-Eurasia Information NetworkTEIN (2004-2007)

Circuit between KOREN(Korea) and RENATER(France). Circuit between KOREN(Korea) and RENATER(France). AP Beneficiaries: AP Beneficiaries:

China, Indonesia, Malaysia, Philippines, Thailand, VietnamChina, Indonesia, Malaysia, Philippines, Thailand, Vietnam (Non-beneficiaries: (Non-beneficiaries: Brunei, Japan, Korea, SingaporeBrunei, Japan, Korea, Singapore EU partners: EU partners: NRENs of France, Netherlands, UKNRENs of France, Netherlands, UK The scope expanded to South-East Asia and China recently. The scope expanded to South-East Asia and China recently. Upgraded to 34 Mbps in 11/2003. Upgrade to 155Mbps planned Upgraded to 34 Mbps in 11/2003. Upgrade to 155Mbps planned

12M Euro EU Funds12M Euro EU Funds Coordinating PartnerCoordinating Partner

DANTE DANTE Direct EU-AP Link; Direct EU-AP Link;

Other Links go Other Links go Across the USAcross the US

APAN China ConsortiumAPAN China Consortium Has been established in 1999.  The China Education and Research Has been established in 1999.  The China Education and Research

Network (Network (CERNETCERNET), the Natural Science Foundation of China ), the Natural Science Foundation of China Network (Network (NSFCNETNSFCNET) and the China Science and Technology ) and the China Science and Technology Network (Network (CSTNETCSTNET) are the main three advanced networks. ) are the main three advanced networks.

CERNet

2.5 Gbps

NSFCnetTsinghua --- Tsinghua UniversityPKU --- Peking UniversityNSFC --- Natural Science Foundation of ChinaCAS --- China Academy of SciencesBUPT --- Beijing Univ. of Posts and Telecom.BUAA --- Beijing Univ. of Aero- and Astronautics

GLORIAD: Global Optical Ring (US-Ru-Cn; Korea Now Full Partner )

GLORIAD: Global Optical Ring (US-Ru-Cn; Korea Now Full Partner )

Also Important for Intra-Russia Connectivity;Education and Outreach

DOE: ITER Distributed Ops.;Fusion-HEP Cooperation

NSF: Collaboration of ThreeMajor R&E Communities

Aug. 8 2004: P.K. Young, Korean IST

Advisor to President Announces

Korea Joining GLORIAD TEIN gradually to 10G, connected to GLORIAD Asia Pacific Info. Infra- Structure (1G) will be

backup net to GLORIAD

GLORIAD and HENP Example:GLORIAD and HENP Example:Network Needs of IHEP Beijing Network Needs of IHEP Beijing

ICFA SCIC Report: Appendix 18, on Network Needs ICFA SCIC Report: Appendix 18, on Network Needs for HEP in China (See for HEP in China (See http://cern.ch/icfa-scichttp://cern.ch/icfa-scic))

““IHEP is working with the Computer Network Information Center (CNIC) and other universities and institutes to build Grid applications for the experiments. The computing resources and storage management systems are being built or upgraded in the Institute. IHEP has a 100 Mbps link to CNIC, so it is quite easy

to connect to GLORIAD and the link could be upgraded as needed.” Prospective Network Needs for IHEP Bejing

Experiment Year 2004-2005 Year 2006 and on

LHC/LCG 622Mbps 2.5Gbps

BES 100Mbps 155Mbps

YICRO 100Mbps 100Mbps

AMS 100Mbps 100Mbps

Others 100Mbps 100Mbps

Total (Sharing) 1Gbps 2.5Gbps

Role of Sciences in Information Society. Palexpo, Geneva 12/2003

Role of Sciences in Information Society. Palexpo, Geneva 12/2003

Demos at the CERN/Caltech Demos at the CERN/Caltech RSIS Online StandRSIS Online Stand

Advanced network and Advanced network and Grid-enabled analysis Grid-enabled analysis

Monitoring very large scale Grid Monitoring very large scale Grid farms with MonALISA farms with MonALISA

World Scale multisite multi-protocol World Scale multisite multi-protocol videoconference with VRVSvideoconference with VRVS

(Europe-US-Asia-South America) (Europe-US-Asia-South America) Distance diagnosis and surgery using Distance diagnosis and surgery using

Robots with “haptic” feedback Robots with “haptic” feedback (Geneva-Canada)(Geneva-Canada)

Music Grid: live performances with Music Grid: live performances with bands at St. John’s, Canada and the bands at St. John’s, Canada and the Music Conservatory of Geneva on Music Conservatory of Geneva on stagestage

VRVS 37k hostsVRVS 37k hosts106 Countries106 Countries

2-3X Growth/Year2-3X Growth/Year

Achieving throughput User can’t achieve throughput available (Wizard gap)

TCP Stack, End-System and/or Local, Regional, Nat’l Network Issues

Big step just to know what is achievable(e.g. 7.5 Gbps over 16 kkm Caltech-CERN)