HENP Networks and Grids for Global Virtual Organizations
HENP Networks and Grids for Global Virtual Organizations. Harvey B. Newman California Institute of Technology TIP2004, Internet2 HENP WG Session January 25, 2004. The Challenges of Next Generation Science in the Information Age. Flagship Applications - PowerPoint PPT Presentation
HENP Networks and Grids for Global Virtual Organizations Harvey B. Newman California Institute of Technology TIP2004, Internet2 HENP WG Session January 25, 2004Computing Model Progress CMS Internal Review of Software and ComputingThe Challenges of Next Generation Science in the Information AgeFlagship Applications High Energy & Nuclear Physics, AstroPhysics Sky Surveys: TByte to PByte block transfers at 1-10+ Gbps eVLBI: Many real time data streams at 1-10 Gbps BioInformatics, Clinical Imaging: GByte images on demandHEP Data Example: From Petabytes in 2003, ~100 Petabytes by 2007-8, to ~1 Exabyte by ~2013-5.Provide results with rapid turnaround, coordinating large but limited computing and data handling resources, over networks of varying capability in different world regionsAdvanced integrated applications, such as Data Grids, rely on seamless operation of our LANs and WANsWith reliable, quantifiable high performancePetabytes of complex data explored and analyzed by 1000s of globally dispersed scientists, in hundreds of teamsComputing Model Progress CMS Internal Review of Software and ComputingFirst Beams: April 2007Physics Runs: from Summer 2007TOTEMpp, general purpose; HILHCb: B-physicsALICE : HI pp s =14 TeV L=1034 cm-2 s-1 27 km Tunnel in Switzerland & FranceLarge Hadron Collider (LHC) CERN, Geneva: 2007 Start CMSATLASAtlasComputing Model Progress CMS Internal Review of Software and ComputingFour LHC Experiments: The Petabyte to Exabyte ChallengeATLAS, CMS, ALICE, LHCB Higgs + New particles; Quark-Gluon Plasma; CP Violation Tens of PB 2008; To 1 EB by ~2015 Hundreds of TFlops To PetaFlops 6000+ Physicists & Engineers; 60+ Countries; 250 InstitutionsComputing Model Progress CMS Internal Review of Software and ComputingLHC: Higgs Decay into 4 muons (Tracker only); 1000X LEP Data Rate109 events/sec, selectivity: 1 in 1013 (1 person in a thousand world populations)Computing Model Progress CMS Internal Review of Software and ComputingLHC Data Grid Hierarchy:Developed at CaltechTier 1Online System CERN Center PBs of Disk; Tape RobotFNAL CenterIN2P3 Center INFN Center RAL Center InstituteInstituteInstituteInstitute Workstations~100-1500 MBytes/sec2.5-10 Gbps0.1 to 10 GbpsTens of Petabytes by 2007-8. An Exabyte ~5-7 Years later.Physics data cache~PByte/sec~10 Gbps~2.5-10 GbpsTier 0 +1Tier 3Tier 4Tier 2ExperimentCERN/Outside Resource Ratio ~1:2 Tier0/( Tier1)/( Tier2) ~1:1:1Emerging Vision: A Richly Structured, Global Dynamic SystemComputing Model Progress CMS Internal Review of Software and ComputingBandwidth Growth of Intl HENP Networks (US-CERN Example) Rate of Progress >> Moores Law. (US-CERN Example) 9.6 kbps Analog (1985) 64-256 kbps Digital (1989 - 1994) [X 7 27] 1.5 Mbps Shared (1990-3; IBM) [X 160] 2 -4 Mbps (1996-1998) [X 200-400] 12-20 Mbps (1999-2000) [X 1.2k-2k] 155-310 Mbps (2001-2) [X 16k 32k] 622 Mbps (2002-3) [X 65k] 2.5 Gbps (2003-4) [X 250k] 10 Gbps (2005) [X 1M]A factor of ~1M over a period of 1985-2005 (a factor of ~5k during 1995-2005) HENP has become a leading applications driver, and also a co-developer of global networks; Computing Model Progress CMS Internal Review of Software and ComputingHEP is Learning How to Use Gbps Networks Fully: Factor of ~50 Gain in Max. Sustained TCP Thruput in 2 Years, On Some US+Transoceanic Routes 9/01 105 Mbps 30 Streams: SLAC-IN2P3; 102 Mbps 1 Stream CIT-CERN 5/20/02 450-600 Mbps SLAC-Manchester on OC12 with ~100 Streams 6/1/02 290 Mbps Chicago-CERN One Stream on OC12 9/02 850, 1350, 1900 Mbps Chicago-CERN 1,2,3 GbE Streams, 2.5G Link 11/02 [LSR] 930 Mbps in 1 Stream California-CERN, and California-AMS FAST TCP 9.4 Gbps in 10 Flows California-Chicago 2/03 [LSR] 2.38 Gbps in 1 Stream California-Geneva (99% Link Utilization) 5/03 [LSR] 0.94 Gbps IPv6 in 1 Stream Chicago- Geneva TW & SC2003: 5.65 Gbps (IPv4), 4.0 Gbps (IPv6) in 1 Stream Over 11,000 km *Computing Model Progress CMS Internal Review of Software and ComputingFAST TCP: Baltimore/Sunnyvale 1 flow 2 flows 7 flows 9 flows 10 flowsAverage utilization95%92%90%90%88%Measurements 11/02Std Packet SizeUtilization averaged over > 1hr4000 km PathFast convergence to equilibriumRTT estimation: fine-grain timerDelay monitoring in equilibriumPacing: reducing burstinessFair Sharing Fast Recovery8.6 Gbps;21.6 TB in 6 Hours9G10GComputing Model Progress CMS Internal Review of Software and ComputingFall 2003: Transatlantic Ultraspeed TCP Tranfers Throughput Achieved: X50 in 2 yearsJuniper, HP Level(3) Telehouse Terabyte Transfers by the Caltech-CERN Team: Nov 18: 4.00 Gbps IPv6 Geneva-Phoenix (11.5 kkm) Oct 15: 5.64 Gbps IPv4 Palexpo-L.A. (10.9 kkm) Across Abilene (Internet2) Chicago-LA, Sharing with normal network traffic Peaceful Coexistence with a Joint Internet2-Telecom World VRVS Videoconference Nov 19: 23+ Gbps TCP: Caltech, SLAC, CERN, LANL, UvA, ManchesterComputing Model Progress CMS Internal Review of Software and ComputingHENP Major Links: Bandwidth Roadmap (Scenario) in GbpsContinuing the Trend: ~1000 Times Bandwidth Growth Per Decade; We are Rapidly Learning to Use Multi-Gbps Networks DynamicallyComputing Model Progress CMS Internal Review of Software and ComputingYearProductionExperimentalRemarks20010.155 0.622-2.5SONET/SDH20020.6222.5SONET/SDHDWDM; GigE Integ.20032.510 DWDM; 1 + 10 GigEIntegration2005102-4 X 10 ( Switch;( Provisioning20072-4 X 10~10 X 10; 40 Gbps1st Gen. ( Grids2009~10 X 10or 1-2 X 40 ~5 X 40 or~20-50 X 1040 Gbps (Switching2011~5 X 40 or~20 X 10~25 X 40 or ~100 X 10 2nd Gen ( GridsTerabit Networks2013~Terabit~MultiTbps~Fill One Fiber HENP Lambda Grids:Fibers for PhysicsProblem: Extract Small Data Subsets of 1 to 100 Terabytes from 1 to 1000 Petabyte Data StoresSurvivability of the HENP Global Grid System, with hundreds of such transactions per day (circa 2007) requires that each transaction be completed in a relatively short time. Example: Take 800 secs to complete the transaction. Then Transaction Size (TB) Net Throughput (Gbps) 1 10 10 100 100 1000 (Capacity of Fiber Today)Summary: Providing Switching of 10 Gbps wavelengths within ~2-4 years; and Terabit Switching within 5-8 years would enable Petascale Grids with Terabyte transactions, to fully realize the discovery potential of major HENP programs, as well as other data-intensive research.Computing Model Progress CMS Internal Review of Software and ComputingNational Light Rail FootprintNLRStarting Up NowInitially 4 10 Gb WavelengthsFuture: to 40 10Gb Waves Transition beginning now to optical, multi-wavelength R&E networks. Also Note: XWIN (Germany); IEEAF/GEO plan for dark fiber in EuropeComputing Model Progress CMS Internal Review of Software and ComputingGLIF network 1Q2004:Global Lambda Integrated FacilityDWDM SURFnet lambda service pathIP service path10 Gbit/sSURFnet10 Gbit/sSURFnet10 Gbit/sIEEAF10 Gbit/sDwingelooASTRON/JIVEPragueCzechLight2.5 Gbit/sNSF10 Gbit/sLondonUKLightStockholmNorthernLight2.5 Gbit/s New YorkMANLANTokyoWIDE10 Gbit/s 10Gbit/s10 Gbit/s(2/29 ?)2x10 Gbit/sIEEAF10 Gbit/s2x10 Gbit/s10 Gbit/s2.5 Gbit/s 2.5 Gbit/sTokyoAPANGenevaCERNChicagoAmsterdam(2/29 )Computing Model Progress CMS Internal Review of Software and ComputingAarnet: SXTransport Project in 2004Connect Major Australian Universities to 10 Gbps BackboneTwo 10 Gbps Research Links to the USAarnet/USLIC Collaboration on Net R&D Starting NowComputing Model Progress CMS Internal Review of Software and ComputingGLORIAD: Global Optical Ring (US-Ru-Cn)Little Gloriad (OC3) Launched January 12; to OC192 in 2004Computing Model Progress CMS Internal Review of Software and ComputingGermany: 2003, 2004, 2005GWIN Connects 550 Universities, Labs, Other InstitutionsGWIN: Q4/03GWIN: Q4/04PlanXWIN: Q4/05 (Dark Fiber Option)Computing Model Progress CMS Internal Review of Software and ComputingThe original Computational and Data Grid concepts are largely stateless, open systems: known to be scalable Analogous to the WebThe classical Grid architecture has a number of implicit assumptions The ability to locate and schedule suitable resources, within a tolerably short time (i.e. resource richness) Short transactions with relatively simple failure modesHENP Grids are Data Intensive & Resource-Constrained 1000s of users competing for resources at 100s of sites Resource usage governed by local and global policies Long transactions; some long queuesHENP Stateful, End-to-end Monitored and Tracked Paradigm Adopted in OGSA, Now WS Resource FrameworkClassical, HENP Data Grids, and Now Service-Oriented GridsComputing Model Progress CMS Internal Review of Software and Computing*Caltech GAE TeamThe Grid Analysis Environment (GAE)The GAE: key to success or failure for physics & Grids in the LHC era:100s - 1000s of tasks, with a wide range of computing, data and network resource requirements, and prioritiesComputing Model Progress CMS Internal Review of Software and ComputingThe Move to OGSA and then Managed Integration SystemsIncreased functionality,standardizationTimeCustomsolutionsOpen GridServices ArchGGF: OGSI, (+ OASIS, W3C)Multiple implementations,including Globus ToolkitWeb services + Globus ToolkitDefacto standardsGGF: GridFTP, GSIX.509,LDAP,FTP, App-specificServices~Integrated SystemsStateful; ManagedWeb Serv Resrc FramwkComputing Model Progress CMS Internal Review of Software and ComputingManaging Global Systems: Dynamic Scalable Services ArchitectureMonALISA: http://monalisa.cacr.caltech.eduComputing Model Progress CMS Internal Review of Software and ComputingDynamic Distributed Services Architecture (DDSA)Station Server Services-engines at sites host Dynamic Services Auto-discovering, Collaborative Scalable to thousands of service-InstancesServers interconnect dynamically; form a robust fabric Service Agents: Goal-Oriented, Autonomous, Adaptive Adaptable to Web services: many platforms & working environments (also mobile)Caltech/UPB (Romania)/NUST (Pakistan) CollaborationSee http://monalisa.cacr.caltech.edu http://diamonds.cacr.caltech.eduComputing Model Progress CMS Internal Review of Software and Computing*Caltech GAE TeamGAE ArchitectureAnalysis Clients talk standard protocols to the Grid Services Web Server, a.k.a. the Clarens data/services portal.Simple Web service API allows Analysis Clients (simple or complex) to operate in this architecture.Typical clients: ROOT, Web Browser, IGUANA, COJAC The Clarens portal hides the complexity of the Grid Services from the client, but can expose it in as much detail as reqd for e.g. monitoring.Key features: Global Scheduler, Catalogs, Monitoring, and Grid-wide Execution service. Computing Model Progress CMS Internal Review of Software and Computing*Caltech GAE TeamGAE ArchitectureThe GAE, based on Clarens and Web services, easily allows a Peer-to-Peer configuration to be built, with associated robustness and scalability features.Flexible: allows easy creation, use and management of highly complex VO structures.A typical Peer-to-Peer scheme would involve the Clarens servers acting as Global Peers, that broker GAE client requests among all the Clarens servers available worldwide.Structured Peer-to-PeerComputing Model Progress CMS Internal Review of Software and ComputingUltraLight Collaboration:http://ultralight.caltech.eduIntegrated hybrid experimental network, leveraging Transatlantic R&D network partnerships; packet-switched + dynamic optical paths 10 GbE across US and the Atlantic: NLR, DataTAG, TransLight, NetherLight, UKLight, etc.; Extensions to Japan, Taiwan, BrazilEnd-to-end monitoring; Realtime tracking and optimization; Dynamic bandwidth provisioningAgent-based services spanning all layers of the system, from the optical cross-connects to the applications.Caltech, UF, FIU, UMich, SLAC,FNAL, MIT/Haystack, CERN, UERJ(Rio), NLR, CENIC, UCAID, Translight, UKLight, Netherlight, UvA, UCLondon, KEK, Taiwan Cisco, Level(3) Computing Model Progress CMS Internal Review of Software and ComputingICFA Standing Committee on Interregional Connectivity (SCIC)Created by ICFA in July 1998 in Vancouver ; Following ICFA-NTFCHARGE: Make recommendations to ICFA concerning the connectivity between the Americas, Asia and Europe (and network requirements of HENP)As part of the process of developing these recommendations, the committee should Monitor traffic Keep track of technology developmentsPeriodically review forecasts of future bandwidth needs, and Provide early warning of potential problemsCreate subcommittees when necessary to meet the chargeThe chair of the committee should report to ICFA once per year, at its joint meeting with laboratory directors (Today)Representatives: Major labs, ECFA, ACFA, NA Users, S. AmericaComputing Model Progress CMS Internal Review of Software and Computing ICFA SCIC in 2002-2004A Period of Intense ActivityStrong Focus on the Digital Divide ContinuingFive Reports; Presented to ICFA 2003 See http://cern.ch/icfa-scic Main Report: Networking for HENP [H. Newman et al.] Monitoring WG Report[L. Cottrell] Advanced Technologies WG Report[R. Hughes-Jones, O. Martin et al.]Digital Divide Report [A. Santoro et al.]Digital Divide in Russia Report [V. Ilyin]2004 Reports in Progress; Short Reports on Natl and Regional Network Infrastructures and Initiatives. Presentation to ICFA February 13, 2004Computing Model Progress CMS Internal Review of Software and ComputingSCIC Report 2003 General ConclusionsThe scale and capability of networks, their pervasiveness and range of applications in everyday life, and HENPs dependence on networks for its research, are all increasing rapidly. However, as the pace of network advances continues to accelerate, the gap between the economically favored regions and the rest of the world is in danger of widening. We must therefore work to Close the Digital DivideTo make Physicists from All World Regions Full Partners in Their Experiments; and in the Process of Discovery This is essential for the health of our global experimental collaborations, our plans for future projects, and our field.Computing Model Progress CMS Internal Review of Software and ComputingWork on the Digital Divide:Several PerspectivesWork on Policies and/or Pricing: pk, in, br, cn, SE Europe, Share Information: Comparative Performance and Pricing Find Ways to work with vendors, NRENs, and/or GovtsExploit Model Cases: e.g. Poland, Slovakia, Czech RepublicInter-Regional Projects South America: CHEPREO (US-Brazil); EU @LIS ProjectGLORIAD, Russia-China-US Optical RingVirtual SILK Highway Project (DESY): FSU satellite linksHelp with Modernizing the Infrastructure Design, Commissioning, DevelopmentProvide Tools for Effective Use: Monitoring, CollaborationWorkshops and Tutorials/Training SessionsFor Example: Digital Divide and HEPGrid Workshop, UERJ Rio, February 2004Participate in Standards Development; Open ToolsAdvanced TCP stacks; Grid systemsComputing Model Progress CMS Internal Review of Software and ComputingICTP 2nd Open Round Table on Developing Countries Access to Scientific Information STATEMENT: AFFORDABLE ACCESS TO THE INTERNETFOR RESEARCH AND LEARNING Scholars from across the world meeting at the Abdus Salam International Centre for Theoretical Physics (ICTP) in Trieste [10/2003] were concerned to learn of the barrier to education and research caused by the high cost of Internet access in many countries. The Internet enables the use of content which is vital for individuals and for institutions engaged in teaching, learning and research. In many countries use of the Internet is severely restricted by the high telecommunications cost, leading to inequality in realising the benefits of education and research. Research staff and students in countries with liberal telecommunications policies favouring educational use are gaining social and economic advantage over countries with restrictive, high-cost policies. The potential benefits of access to the Internet are not available to all. The signatories to this message invite scholars in every country to join them in expressing concern to governments and research funding agencies at the effect of high telecommunications costs upon individuals and institutions undertaking teaching, learning and research. The situation in many countries could be improved through educational discounts on normal telecommunications costs, or through the lifting of monopolies. It is for each country to determine its own telecommunications policies but the need for low-cost access to the Internet for educational purposes is a need which is common to the whole of humankind.Computing Model Progress CMS Internal Review of Software and Computing History - Throughput Quality Improvements from US Bandwidth of TCP < MSS/(RTT*Sqrt(Loss)) (1)(1) Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Matthis, Semke, Mahdavi, Ott, Computer Communication Review 27(3), July 199760% annual improvement Factor ~100/10 yrProgress: but the Digital Divide is Being Maintained1.5-8 Year LagS.E. Europe, Russia: Catching UpLat Am., MidEast, China, Africa:Keeping UpIndia: Falling BehindComputing Model Progress CMS Internal Review of Software and ComputingCurrent State June 2003 (Max. Throughput in Mbps)Within region performance improving: E.g. Ca/US-NA, Hu-SE Eu, Eu-Eu, Jp-E Asia, Au-Au, Ru-RuAfrica, Caucasus, Central & S. Asia all badBad < 200kbits/s < DSL Acceptable > 200, < 1000kbits/sGood > 1000kbits/sComputing Model Progress CMS Internal Review of Software and ComputingDAI: State of the WorldComputing Model Progress CMS Internal Review of Software and ComputingDigital Access Index Top TenComputing Model Progress CMS Internal Review of Software and ComputingDAI: State of the WorldComputing Model Progress CMS Internal Review of Software and ComputingComputing Model Progress CMS Internal Review of Software and Computing APAN Link Information 2004.1.05 email@example.comCountriesNetworkBandwidth(Mbps)AUP/RemarkAU-USAARNet310 to 2 X 10 Gbps R&E + CommodityCN-HKCERNET/HARNET2 R&ECN-HKCSTNET155 R&ECN-JPCERNET10 R&ECN-JPCERNET45 Native IPv6CN-UKCERNET45 R&ECN-USCERNET10 Research CN-USCERNET200 R&ECN-USCSTNET155 R&EHK-USHARNET45 R&EHK-TWHARNET/TANET10 R&EJP-ID AI3(ITB) 2/1.5 R&EJP-KR APII 2Gbps R&EJP-MY AI3(USM) 1.5/0.5 R&EJP-PH AI3(ASTI) 1.5/0.5 R&EJP-PH MAFFIN2 Research JP-SG AI3(SICU) 1.5/0.5 R&EJP-TH AI3(AIT) 1.5/0.5 R&EJP-TH SINET(ThaiSarn)2 R&EJP-USTransPac5 Gbps R&EJP-VN AI3(IOIT) 1.5/0.5 R&EComputing Model Progress CMS Internal Review of Software and Computing APAN Link Information 2004.1.05 firstname.lastname@example.orgCountriesNetworkBandwidth(Mbps)AUP/RemarkKR-FRKOREN/RENATER34 Research (TEIN) KR-SG APII8 R&EKR-US KOREN/KREONet2 155 R&ELK-JPLEARN3 R&E MY-SGNRG/SICU2 Experiment (Down)SG-US SingaREN 90 RÐ-USUninet155 R&E TW-HKASNET/TANET2155 R&ETW-JPASNET/TANET2155 R&ETW-USASNET/TANET25 Gbps R&E(TW)-US-UKASNET/TANET2155 R&ENon APAN Links CountriesNetworkBandwidth(Mbps)AUPIN-US/UKERNET16 R&E(JP)-US-EUSINET155 R&E / No TransitJP-USSINET5 Gbps R&E / No TransitComputing Model Progress CMS Internal Review of Software and ComputingBrazil: RNP in Early 2004Computing Model Progress CMS Internal Review of Software and ComputingDai Davies SERENATE Workshop Feb. 2003Computing Model Progress CMS Internal Review of Software and ComputingInternational Connectivity Costs in theDifferent European Market Segments Market segment Number of CountriesCost RangeLiberal Market with transparent pricing 81-1.4Liberal Market with less transparent pricing structure 71.8-3.3Emerging Market without transparent pricing37.5-7.8Traditional Monopolist market 918-39Virtual Silk Highway:The Silk CountriesComputing Model Progress CMS Internal Review of Software and ComputingVirtual SILK HighwayArchitectural OverviewHub Earth Station at DESY with access to the European NRENs and the Internet via GEANTProviding International Internet access directlyNational Earth Station at each Partner siteOperated by DESY, providing international access Additional earth stations from other sources none yetSCPC up-link, common down-link, using DVBRouters for each Partner siteLinked on one side to the Satellite ChannelOn the other side to the NRENComputing Model Progress CMS Internal Review of Software and ComputingBandwidth Plan as of 3/03Computing Model Progress CMS Internal Review of Software and ComputingProgress in Slovakia 2002-2004(January 2004)Computing Model Progress CMS Internal Review of Software and ComputingVRVS on WindowsVRVS (Version 3)Meeting in 8 Time Zones80+ Reflectors24.5k hosts worldwideUsers in 99 Countries Computing Model Progress CMS Internal Review of Software and Computing SERENATE is the name of a series of strategic studies into the future of research and education networking in Europe, addressing the local (campus networks), national (national research & education networks), European and intercontinental levels. The SERENATE studies bring together the research and education networks of Europe, national governments and funding bodies, the European Commission, traditional and "alternative" network operators, equipment manufacturers, and the scientific and education community as the users of networks and services. Study into European Research and Education Networking as Targeted by eEuropewww.serenate.orgSummary and Conclusions by D.O. Williams, CERN Computing Model Progress CMS Internal Review of Software and ComputingOptics and Fibres[Message to NRENs; or Natl Initiatives]If there is one single technical lesson from SERENATE it is that transmission is moving from the electrical domain to optical. The more you look at underlying costs the more you see the need for users to get access to fibre.When theres good competition users can still lease traditional communications services (bandwidth) on an annual basis. But: Without enough competition prices go through the roof.A significant divide exists inside Europe with the worst countries [Macedonia, B-H, Albania, etc.] 1000s of times worse off than the best. Also many of the 10 new EU members are ~5X worse off than the 15 present members.Our best advice has to be if youre in a mess, you must get access to fibre.Also try to lobby politicians to introduce real competition.In Serbia still a full telecoms monopoly the two ministers talked and the research community was given a fibre pair all around Serbia !Computing Model Progress CMS Internal Review of Software and ComputingNEWS: Bulletin: ONE TWO WELCOME BULLETIN General Information Registration Travel Information Hotel RegistrationParticipant List How to Get UERJ/Hotel Computer AccountsUseful Phone Numbers ProgramContact us: Secretariat Chairmen CLAF CNPQ FAPERJ UERJ SPONSORSHEPGRID and Digital Divide Workshop UERJ, Rio de Janeiro, Feb. 16-20 2004Theme: Global Collaborations, Grids and Their Relationship to the Digital DivideICFA, understanding the vital role of these issues for our fields future, commissioned the Standing Committee on Inter-regional Connectivity (SCIC) in 1998, to survey and monitor the state of the networks used by our field, and identify problems. For the past three years the SCIC has focused on understanding and seeking the means of reducing or eliminating the Digital Divide, and proposed in ICFA that these issues, as they affect our field of High Energy Physics, be brought to our community for discussion. This led to ICFAs approval, in July 2003, of the Digital Divide and HEP Grid Workshop. More Information: http://www.uerj.br/lishep2004Computing Model Progress CMS Internal Review of Software and ComputingNetworks, Grids and HENPNetwork backbones and major links used by HENP experiments are advancing rapidly To the 10 G range in < 2 years; much faster than Moores Law Continuing a trend: a factor ~1000 improvement per decade; a new DOE and HENP Roadmap Transition to a community-owned and operated infrastructure for research and education is beginning with (NLR, USAWaves) HENP is learning to use long distance 10 Gbps networks effectively 2002-2003 Developments: to 5+ Gbps flows over 11,000 kmRemoving Regional, Last Mile, Local Bottlenecks and Compromises in Network Quality are now On the critical path, in all world regionsDigital Divide: Network improvements are especially needed in SE Europe, So. America; SE Asia, and Africa Work in Concert with Internet2, Terena, APAN, AMPATH; DataTAG, the Grid projects and the Global Grid ForumComputing Model Progress CMS Internal Review of Software and ComputingSome Extra Slides FollowComputing Model Progress CMS Internal Review of Software and ComputingICFA and International Networking ICFA Statement on Communications in Intl HEP Collaborations of October 17, 1996 See http://www.fnal.gov/directorate/icfa/icfa_communicaes.htmlICFA urges that all countries and institutions wishing to participate even more effectively and fully in international HEP Collaborations should: Review their operating methods to ensure they are fully adapted to remote participation Strive to provide the necessary communications facilities and adequate international bandwidthComputing Model Progress CMS Internal Review of Software and ComputingNREN Core Network Size (Mbps-km):http://www.terena.nl/compendium/2002Logarithmic Scale1k100k100100M10M1M10kRoItPlGrIrUkrHuCzEsNlFiChLaggingIn TransitionLeadingAdvancedComputing Model Progress CMS Internal Review of Software and ComputingNetwork Readiness Index:How Ready to Use Modern ICTs [*]?Network Readiness IndexEnvironmentReadinessUsageMarketInfrastructurePolitical/RegulatoryIndividual ReadinessGovt ReadinessBusiness ReadinessIndividual UsageGovt UsageBusiness Usage(FI)(FI)(SG)(US)(IC)(SG)(US)(SG)(US)(FI)(FI)(DE)(KR)( ): Which Country is First From the 2002-2003 Global InformationTechnology Report. See http://www.weforum.org Computing Model Progress CMS Internal Review of Software and Computing PINGER History Loss QualityFewer sites have very poor to dreadful performanceMore have good performance (< 1% Loss)BUT Throughput vs Net Readiness Index NRI from Center for Intl Development, Harvard http://www.cid.harvard.edu/cr/pdf/gitrr2002_ch02.pdfImproved correlation (0.21 0.41) by Using derived throughput ~ MSS / (RTT * sqrt(loss)); fit an exponentialInteresting Outliers: Slovakia, Hungary, Portugal. LithuaniaInternet for all focusA&R focusNRI TopsFinland 5.92US 5.79Singapore5.74Sweden 5.58Iceland 5.51Canada 5.44UK5.35Denmark 5.33Taiwan5.31Germany5.29Netherlnd 5.28Israel 5.22Switzland 5.18Korea 5.10Computing Model Progress CMS Internal Review of Software and ComputingUltraLight http://ultralight.caltech.edu Serving the major LHC experiments; developments broadly applicable to other data-intensive programs Hybrid packet-switched and circuit-switched, dynamically managed optical network Global services for system management Trans-US wavelength riding on NLR: LA-SNV-CHI-JAX Leveraging advanced research & production networks USLIC/DataTAG, SURFnet/NLlight, UKLight, Abilene, CA*net4 Dark fiber to CIT, SLAC, FNAL, UMich; Florida Light Rail Intercontl extensions: Rio de Janeiro, Tokyo, Taiwan Flagship Applications with a diverse traffic mix HENP: TByte to PByte block data transfers at 1-10+ Gbps eVLBI: Real time data streams at 1 to several GbpsComputing Model Progress CMS Internal Review of Software and ComputingBy I. Legrand (Caltech) et al.Monitors Clusters, NetworksAgent-based Dynamic information / resource discovery mechanismsImplemented in Java/Jini; SNMPWDSL / SOAP with UDDIGlobal System Optimizations > 100 Sites and GrowingBeing deployed in Abilene; through the Internet2 E2EPiMonALISA (Java) 3D InterfaceMonaLisa: A Globally Scalable Grid Monitoring SystemComputing Model Progress CMS Internal Review of Software and ComputingPingER BenefitsMeasures analyzes & reports round-trip times, losses, availability, throughput ...Covers 75+ countries (99% of Internet connected population)Low impact on network 8yrs) and near real-time informationHow bad is performance to various regions, rank countries?Trends: who is catching up, falling behind, is progress being made?Compare vs. economic, development indicators etc.Useful for troubleshooting, setting expectations; presenting to policy makers, funding bodiesComputing Model Progress CMS Internal Review of Software and ComputingCountries MonitoredUsed to monitorOnly 1 hostNeed > 1 host to reduce anomaliesComputing Model Progress CMS Internal Review of Software and ComputingPingER SummaryPerformance is improving all overPerformance to developed countries are orders of magnitude better than to developing countriesPoorer regions 5-10 years behindPoorest regions Africa, Caucasus, Central & S. AsiaSome regions are:Catching up: SE Europe, RussiaKeeping up:Latin America, Mid East, China Falling further behind:E.g, India, AfricaComputing Model Progress CMS Internal Review of Software and ComputingUser RequirementsIn ALL countries and in ALL disciplines researchers are eagerly anticipating improved networking tools. There is no divide on the demand-side. Sciences, such as particle physics, which make heavy use of advanced networking, must help to break down any divide on the supply-side, or else declare themselves elitist and irrelevant to researchers in essentially all developing countries. Computing Model Progress CMS Internal Review of Software and ComputingConnectivity pricing and competitionIn some locations the price of connectivity is (really) unreasonably high Linked (obviously) to how competitive the market isStrong competition on routes between various key European cities, and between major national centresLess competition effectively none as you move to countries with de facto monopoly or simply to parts of countries where operators see little reason to invest.While some expensive routes are where you would expect, others are much more surprising (at first sight), like Canterbury and Lancaster (UK) and parts of Brittany (F)Computing Model Progress CMS Internal Review of Software and ComputingUnderstanding transmission costs and DIY solutionsOwn trenching only makes sense in very special cases. Say 1-30 km. Even then look for partners.Maybe useful (as a threat) over longer distances in countries with crazy pricingNow possible to lease (short- or long-term) fibres on many routes in Europe [0.5 to 2 KEuro/km Typ.]Transmission costs jump at ~200 km [below which you can operate with Nothing In Line (NIL), above which you need amplifiers] and ~800 km [above which you need signal regenerators]Possibly leading to some new approaches in GEANT-2 implementationComputing Model Progress CMS Internal Review of Software and ComputingAN INTERESTING STUDY MADE BY FUNDAO GETULIO VARGAS - BRAZIL.; REPORTED by A. Santoro (UERJ) July 11 SCIC MeetingComputing Model Progress CMS Internal Review of Software and ComputingUntil 2001, only 12,46% of Brazilians had Access to a PC and 8,31% to InternetComputing Model Progress CMS Internal Review of Software and ComputingWith a Very Non-Uniform Distribution. For Example: Concentration in Parts of Rio de JaneiroComputing Model Progress CMS Internal Review of Software and Computing% Having :PC - Internet - % Total Pop. BRAZIL: Access to a PC, and to the Internet Vs. Yrs. of Study 0 YEARS TO 4 YEARS 4 TO 8 YEARS 8 TO 12 YEARS MORE THAN 12 YEARS~1/3 of those with Access to a PC and to the Internet is concentrated in the hands of the 6% of the population who graduated from high school Computing Model Progress CMS Internal Review of Software and ComputingJensen, ICTPTyp. 0-7 bps Per PersonComputing Model Progress CMS Internal Review of Software and Computing Limited by many externalsystemic factors: Electricity;Import Duties;Education; Trade restrictions Jensen, ICTPProgress in Africa ?Computing Model Progress CMS Internal Review of Software and ComputingComputing Model Progress CMS Internal Review of Software and ComputingComputing Model Progress CMS Internal Review of Software and ComputingDAI: State of the WorldComputing Model Progress CMS Internal Review of Software and ComputingDAI: State of the WorldComputing Model Progress CMS Internal Review of Software and ComputingDAI: State of the WorldComputing Model Progress CMS Internal Review of Software and ComputingCalifornia Institute of TechnologyComputing Model Progress CMS Internal Review of Software and ComputingApplication Empowerment of Global Systems: Key Role of HENPEffective use of networks is vital for the existence and daily operation of Global CollaborationsPhysicists today face the greatest challenges in terms of Data intensiveness; volume and complexityDistributed computation and storage resourcesGlobal dispersion of many cooperating research teams Physicists and computer scientists have become leading co-developers of networks and global systems Building on a tradition of building next-generation systems that harness new technologies in the service of scienceMission OrientationTackle the hardest problems, to enable the science, maintaining a years-long commitmentBroad Applicability to Other Fields of Research, SocietyComputing Model Progress CMS Internal Review of Software and ComputingNext Generation Grid Challenges: Workflow Management & OptimizationScaling to Handle Thousands of Simultaneous RequestsIncluding the Network as a Dynamic, Managed ResourceCo-Scheduled with Computing and StorageMaintaining a Global View of Resources and System StateEnd-to-end MonitoringAdaptive Learning: New paradigms for optimization, problem resolution Balancing Policy Against Moment-to-moment Capability High Levels of Usage of Limited Resources Versus Better Turnaround Times for Priority TasksStrategic Workflow Planning; Strategic RecoveryAn Integrated User Environment User-Grid Interactions; Progressive AutomationEmerging Strategies and GuidelinesComputing Model Progress CMS Internal Review of Software and ComputingGrid Enabled Analysis: View of a Collaborative Desktop Building the GAE is the Acid Test for Grids; and is crucial for next-generation experiments at the LHC Large, Diverse, Distributed Community of users Support hundreds to thousands of analysis tasks, shared among dozens of sites Widely varying task requirements and prioritiesNeed Priority Schemes, robust authentication and SecurityRelevant to the future needs of research and industryPDAROOTClarensExternal ServicesMonaLisaBrowserIguanaVO ManagementAuthenticationAuthorizationLoggingKey EscrowFile AccessShellStorage Resource Broker CMS ORCA/COBRACluster SchedulersATLAS DIALGriphyn VDTMonaLisa MonitoringComputing Model Progress CMS Internal Review of Software and ComputingCurrent ProblemsSiting of the Earth Station - UzbekistanAUPs ArmeniaLicence - ArmeniaExistence of NREN TurkmenistanShortage of Bandwidth GeorgiaNumber of Earth Stations KazakhstanMarginal transmitters putting in amplifiersComputing Model Progress CMS Internal Review of Software and ComputingNext Generation Grid Challenges: Workflow Management & OptimizationScaling to Handle Thousands of Simultaneous RequestsIncluding the Network as a Dynamic, Managed ResourceCo-Scheduled with Computing and StorageMaintaining a Global View of Resources and System StateEnd-to-end MonitoringAdaptive Learning: New paradigms for optimization, problem resolution Balancing Policy Against Moment-to-moment Capability High Levels of Usage of Limited Resources Versus Better Turnaround Times for Priority TasksStrategic Workflow Planning; Strategic RecoveryAn Integrated User Environment User-Grid Interactions; Progressive AutomationEmerging Strategies and GuidelinesComputing Model Progress CMS Internal Review of Software and ComputingAbilene - Upgrade Completed!Computing Model Progress CMS Internal Review of Software and ComputingSC2003 Bandwidth ChallengeComputing Model Progress CMS Internal Review of Software and ComputingComputing Model Progress CMS Internal Review of Software and Computing Available fiber topologyComputing Model Progress CMS Internal Review of Software and ComputingLeading & Emerging Regional Optical InitiativesCalifornia (CALREN)Colorado (FRGP/BRAN)Connecticut (Connecticut Education Network)Florida (Florida LambdaRail)Indiana (I-LIGHT)Illinois (I-WIRE)Maryland, D.C. & northern Virginia (MAX)MichiganMinnesotaNew York + New England region (NEREN)North Carolina (NC LambdaRail)Ohio (Third Frontier Network)Oregon Rhode Island (OSHEAN)SURA Crossroads (southeastern U.S.)TexasUtahWisconsinComputing Model Progress CMS Internal Review of Software and ComputingSlide 7 - This diagram shows how the client access the grid services through agrid service host, and a rough idea of how the types of services interact. Onekey point from this slide is that the clients can be as smart or dumb as theapplication desires. They can let the grid service host do all of the work andignore the details of the various services, or they can be more involved in theservice flow. The motivator for hiding the grid services behind a gridservice host is that it allows for simpler and easier to develop clientapplication. The services are still available for smarter clients, however, sothat application developers can have more control over the service flow ifnecessary. Clarens is used as the Grid Service host, providing XMLRPC, SOAP,and HTTP access to the grid services. Clarens also provides authenticationand authorization to allow secure access to services. Factors motivating design choices: HTTP, SOAP, XMLRPC were chosen for transport/communication protocols due to their wide acceptance as standards. The heterogeneous and decentrally administered GAE must use open protocols for communication in order to be widely accepted. This lowers the barriers to entry when introducing new services and GAE grid sites.This slide shows that there is no single instance of a service;services are distributed across many service providing sites. The services atone site can interact with services at another site. Peer to peer is used toprovide dynamic service discovery in order to prevent a single point offailure, and to allow easy integration of new services and service providersinto the system. The term "super peer" is used to describe the naturaltendency for the system to direct more requests to peers that have moreresource availability (disk space, bandwidth, computing nodes); that is, notall peers are created equal and the more resource-affluent peers will tend tohandle more of the grid-wide load than the resource-limited peers. P2Pcapabilities are being added to the Clarens service servers, as an addition to theexisting security and service hosting capabilities. This makes Clarens a verykey component of the CAIGEE architecture. Factors motivating design choices: P2P technologies are widely used for creating ad-hoc and dynamic groups of resource providers (file sharing). While still fairly young, P2P has proven that it is capable of scaling to the size that the GAE aims for.Note this is measured from ESnet Labs in US so biased towards US sitesLines are medians for many measured pairs, so half pairs have better than shownSE Europe=MK, RO, MD, BU, AL, SL, CRThis is the NRI 2002-3 data. The 2002-3 data can be found via http://www.weforum.org/site/homepublic.nsf/Content/Initiatives+subhomeThe NRI is developed in depth in the World Economic Forum "Global Information Technology Report" 2002-3. The WEF takes the Harvard CID definition as a basis. Networked Readiness Index (NRI), a major international assessment of countries capacity to exploit the opportunitiesoffered by ICTs.A communitys potential to participate in the Networked World of the future.NRI: Construct a network use component that measures the extent of current network connectivity, and an enabling factors component that measures a countrys capacity to exploit existing networks and create new ones. Network use is defined by 5 variables related to the quantity and quality of ICT (Information Communication Technology) use. Enabling factors are based on Network access, network policy, networked society and the networked economy.Countries at the bottom right of the right hand graph may be concentrating on Internet access for all, while countries in the upper right may be focussing on excellent academic & research networks.Note for Africa only one host in Uganda. Actually have been adding hosts 5 countries), but there is considerable disparity in performance so as add hosts from less developed countries the aggregate performance measured to Africa is dropping!Need >1 host to reduce anomalies.