Present and Future Networks an HENP Perspective Present and Future Networks an HENP Perspective Harvey B. Newman, Caltech HENP WG Meeting Internet2 Headquarters,

  • Published on
    11-Jan-2016

  • View
    212

  • Download
    0

Transcript

  • Present and Future Networks an HENP Perspective

    Harvey B. Newman, Caltech HENP WG MeetingInternet2 Headquarters, Ann ArborOctober 26, 2001http://l3www.cern.ch/~newman/HENPWG_Oct262001.ppt

    Computing Model Progress CMS Internal Review of Software and Computing

  • Next Generation Networks for ExperimentsMajor experiments require rapid access to event samples and subsets from massive data stores: up to ~500 Terabytes in 2001, Petabytes by 2002, ~100 PB by 2007, to ~1 Exabyte by ~2012.Across an ensemble of networks of varying capabilityNetwork backbones are advancing rapidly to the 10 Gbps range: Gbps end-to-end requirements for data flows will followAdvanced integrated applications, such as Data Grids, rely on seamless transparent operation of our LANs and WANsWith reliable, quantifiable (monitored), high performanceThey depend in turn on in-depth, widespread knowledge of expected throughput Networks are among the Grids basic building blocksWhere Grids interact by sharing common resourcesTo be treated explicitly, as an active part of the Grid designGrids are interactive; based on a variety of networked appsGrid-enabled user interfaces; Collaboratories

    Computing Model Progress CMS Internal Review of Software and Computing

  • LHC Computing Model Data Grid Hierarchy (Ca. 2005)Tier 1Online SystemOffline Farm, CERN Computer Ctr ~25 TIPSFNAL CenterIN2P3 Center INFN Center RAL Center InstituteInstituteInstituteInstitute ~0.25TIPSWorkstations~100 MBytes/sec~2.5 Gbps100 - 1000 Mbits/secPhysicists work on analysis channelsEach institute has ~10 physicists working on one or more channelsPhysics data cache~PByte/sec~2.5 Gbits/sec~2.5 GbpsTier 0 +1Tier 3Tier 4Tier 2ExperimentCERN/Outside Resource Ratio ~1:2 Tier0/( Tier1)/( Tier2) ~1:1:1

    Computing Model Progress CMS Internal Review of Software and Computing

  • Baseline BW for the US-CERN Transatlantic Link: TAN-WG (DOE+NSF)Plan: Reach OC12 Baseline in Spring 2002; then 2X Per Year

    Computing Model Progress CMS Internal Review of Software and Computing

    Chart6

    20

    155

    310

    622

    1250

    2500

    5000

    US-CERN Bandwith

    Years

    Bandwdth in Mbps

    Chart1

    310

    622

    1250

    2500

    5000

    10000

    Bandwidth (Mbps)

    Link Bandwidth (Mbps)

    Sheet1

    FY1999FY2000FY2001FY2002FY2003FY2004FY2005FY2006

    2015531062212502500500010000

    Sheet2

    Sheet3

  • Transatlantic Net WG (HN, L. Price) Bandwidth Requirements [*][*] Installed BW. Maximum Link Occupancy 50% AssumedThe Network Challenge is Shared by Both Next- and Present Generation Experiments

    2001

    2002

    2003

    2004

    2005

    2006

    CMS

    100

    200

    300

    600

    800

    2500

    ATLAS

    50

    100

    300

    600

    800

    2500

    BaBar

    300

    600

    1100

    1600

    2300

    3000

    CDF

    100

    300

    400

    2000

    3000

    6000

    D0

    400

    1600

    2400

    3200

    6400

    8000

    BTeV

    20

    40

    100

    200

    300

    500

    DESY

    100

    180

    210

    240

    270

    300

    CERN

    BW

    155-310

    622

    1250

    2500

    5000

    10000

  • Total U.S. Internet TrafficSource: Roberts et al., 2001 U.S. Internet TrafficVoice Crossover: August 20004/Year2.8/Year1Gbps1Tbps10Tbps100Gbps10Gbps100Tbps100Mbps1Kbps1Mbps10Mbps100Kbps10Kbps100 bps1 Pbps100 Pbps10 Pbps10 bpsARPA & NSF Data to 96New MeasurementsLimit of same % GDP as VoiceProjected at 4/Year

    Computing Model Progress CMS Internal Review of Software and Computing

  • AMS-IX Internet Exchange Throughput Accelerated Growth in Europe (NL)Hourly Traffic 8/23/013.0 Gbps2.0 Gbps1.0 Gbps0Monthly Traffic 4X Growth from 2000-2001

    Computing Model Progress CMS Internal Review of Software and Computing

  • GriPhyN iVDGL Map Circa 2002-2003 US, UK, Italy, France, Japan, Australia International Virtual-Data Grid Laboratory Conduct Data Grid tests at scale Develop Common Grid infrastructure National, international scale Data Grid tests, leading to managed ops (GGOC) Components Tier1, Selected Tier2 and Tier3 Sites Distributed Terascale Facility (DTF) 0.6 - 10 Gbps networks: US, Europe, transoceanic Possible New Partners Brazil T1 Russia T1 Pakistan T2 China T2

    Computing Model Progress CMS Internal Review of Software and Computing

  • Computing Model Progress CMS Internal Review of Software and Computing

  • Abilene and Other Backbone FuturesAbilene partnership with Qwest extended through 2006Backbone to be upgraded to 10-Gbps in three phases: Complete by October 2003 Detailed Design Being Completed Now GigaPoP Upgrade start in February 2002Capability for flexible provisioning in support of future experimentation in optical networking In a multi- infrastructureOverall approach to the new technical design and business plan is for an incremental, non-disruptive transitionAlso: GEANT in Europe; Super-SINET in Japan; Advanced European national networks (DE, NL, etc.)

    Computing Model Progress CMS Internal Review of Software and Computing

  • TEN-155 and GEANTEuropean A&R Networks 2001-2002GEANT: from 9/01 10 & 2.5 GbpsTEN-155 OC12 CoreProject: 2000 - 2004European A&R Networks are Advancing Rapidly

    Computing Model Progress CMS Internal Review of Software and Computing

  • National Research Networks in JapanSuperSINET Start of operation January 2002 Support for 5 important areas: HEP, Genetics, Nano Technology, Space/Astronomy, GRIDs Provides 10 Gbps IP connection Direct inter-site GbE links Some connections to 10 GbE in JFY2002HEPnet-J Will be re-constructed with MPLS-VPN in SuperSINETIMnet Will be merged into SINET/SuperSINETTokyoOsakaNagoyaInternetOsaka UKyoto UICRKyoto-UNagoya UNIFSNIGIMSU-TokyoNAOU TokyoNII HitotsubashiIPWDM pathIP routerISAS

    Computing Model Progress CMS Internal Review of Software and Computing

  • STARLIGHT: The Next GenerationOptical STARTAPStarted this SummerExisting Fiber: Ameritech, AT&T, Qwest; MFN, Teleglobe, Global Crossing and OthersMain distinguishing features:Neutral location (Northwestern University)40 racks for co-location1/10 Gigabit Ethernet basedOptical switches for advanced experiments GMPLS, OBGP2*622 Mbps ATMs connections to the STAR TAPStarLight, the Optical STAR TAP, is an advanced optical infrastructure and proving ground for network services optimized for high-performance applications. In partnership with CANARIE (Canada), SURFnet (Netherlands), and soon CERN.Developed by EVL at UIC, iCAIR at NWU, ANL/MCS Div.

    Computing Model Progress CMS Internal Review of Software and Computing

  • DataTAG ProjectEU-Solicited Project. CERN, PPARC (UK), Amsterdam (NL), and INFN (IT) Main Aims: Ensure maximum interoperability between US and EU Grid ProjectsTransatlantic Testbed for advanced network research2.5 Gbps wavelength-based US-CERN Link 7/2002 (Higher in 2003)

    Computing Model Progress CMS Internal Review of Software and Computing

  • Daily, Weekly, Monthly and Yearly Statistics on 155 Mbps US-CERN Link 20 - 60 Mbps Used RoutinelyBW Upgrades Quickly Followed by Upgraded Production Use

    Computing Model Progress CMS Internal Review of Software and Computing

  • Throughput Changes with Time Link, route upgrades, factors 3-16 in 12 months Improvements in steps at times of upgrades 8/01: 105 Mbps reached with 30 Streams: SLAC-IN2P3 9/1/01: 102 Mbps reached in One Stream: Caltech-CERNSee http://www-iepm.slac.stanford. edu/monitoring/bulk/ Also see the Internet2 E2E Initiative: http://www.internet2.edu/e2e

    Computing Model Progress CMS Internal Review of Software and Computing

  • Caltech to SLAC on CALREN2A Shared Production OC12 NetworkSLAC: 4 CPU Sun; Caltech: 1 GHz PIII; GigE Interfaces

    Need Large Windows; Multiple streams helpBottleneck bandwidth ~320 Mbps; RTT 25 msec; Window > 1 MB needed for a single stream Results vary by a factor of up to 5 over time; sharing with campus trafficCALREN2

    Computing Model Progress CMS Internal Review of Software and Computing

  • Max. Packet Loss Rates for Given Throughput [Matthis: BW < MSS/(RTT*Loss 0.5)] 1 Gbps LA-CERN Throughput Means Extremely Low Packet Loss ~1E-8 with standard packet sizeAccording to the Equation a single stream with 10 Gbps throughput requires a packet loss rate of 7 X 1E-11 with standard size packets 1 packet lost per 5 hours ! LARGE Windows 2.5 Gbps Caltech-CERN 53 Mbytes Effects of Packet Drop (Link Error) on a 10 Gbps Link: MDAI Halve the Rate: to 5 Gbps It will take ~ 4 Minutes for TCP to ramp back up to 10 Gbps Large Segment Sizes (Jumbo Frames) Could Help, Where Supported Motivation for exploring TCP Variants; Other Protocols

    Computing Model Progress CMS Internal Review of Software and Computing

    Sheet1

    LA-Boston70msec

    MSS (Bytes)15009128

    Thruput MbpsLossLoss

    101E+074E-042E-02

    303E+075E-052E-03

    1001E+084E-062E-04

    3003E+085E-072E-05

    1,0001E+094E-082E-06

    3,0003E+095E-092E-07

    10,0001E+104E-102E-08

    LA-CERN170msec

    MSS (Bytes)15009128

    Thruput MbpsLossLoss

    101E+077E-053E-03

    303E+078E-063E-04

    1001E+087E-073E-05

    3003E+088E-083E-06

    1,0001E+097E-093E-07

    3,0003E+098E-103E-08

    10,0001E+107E-113E-09

    Sheet2

    Sheet3

  • Key Network Issues & Challenges Net Infrastructure Requirements for High ThroughputCareful Router configuration; monitoring Enough Router Horsepower (CPUs, Buffer Space)Server and Client CPU, I/O and NIC throughput sufficientPacket Loss must be ~Zero (well below 0.1%) I.e. No Commodity networksNo Local infrastructure bottlenecks Gigabit Ethernet clear path between selected host pairs To 10 Gbps Ethernet by ~2003TCP/IP stack configuration and tuning is Absolutely Required Large Windows Multiple StreamsEnd-to-end monitoring and tracking of performanceClose collaboration with local and regional network engineering staffs (e.g. router and switch configuration).

    Computing Model Progress CMS Internal Review of Software and Computing

  • Key Network Issues & ChallengesNone of this scales from 0.08 Gbps to 10 GbpsNew (expensive) hardwareThe last mile, and tenth-mile problemFirewall performance; security issuesConcerns The Wizard Gap (ref: Matt Matthis; Jason Lee) RFC2914 and the Network Police; Clever FirewallsNet Infrastructure providers (Local, regional, national, intl) who may or may not want (or feel able) to accommodate HENP bleeding edge usersNew TCP/IP developments (or TCP alternatives) are required for multiuser Gbps links [UDP/RTP ?]

    Computing Model Progress CMS Internal Review of Software and Computing

  • Internet2 HENP WG [*]

    To help ensure that the requiredNational and international network infrastructures (end-to-end)Standardized tools and facilities for high performance and end-to-end monitoring and tracking, andCollaborative systems are developed and deployed in a timely manner, and used effectively to meet the needs of the US LHC and other major HENP Programs, as well as the general needs of our scientific community.To carry out these developments in a way that is broadly applicable across many fieldsForming an Internet2 WG as a suitable framework

    [*] Co-Chairs: S. McKee (Michigan), H. Newman (Caltech); Secy J. Williams (Indiana); With thanks to Rob Gardner (Indiana)http://www.usatlas.bnl.gov/computing/mgmt/lhccp/henpnet/

    Computing Model Progress CMS Internal Review of Software and Computing

  • Network-Related Hard ProblemsQuery Estimation: Reliable Estimate of PerformanceThroughput monitoring, and also ModelingSource and Destination Host & TCP-stack BehaviorPolicy Versus Technical Capability IntersectionStrategies: (New Algorithms) Authentication, Authorization, Priorities and Quotas Across SitesMetrics of PerformanceMetrics of Conformance to PolicyKey Role of Simulation (for Grids as a Whole): Now Casting ?

    Computing Model Progress CMS Internal Review of Software and Computing

  • US CMS Remote Control RoomFor LHCUS CMS will use the CDF/KEK remote control room concept for Fermilab Run II as a starting point. However, we will (1) expand the scope to encompass a US based physics group and US LHC accelerator tasks, and (2) extend the concept to a Global Collaboratory for realtime data acquisition + analysis

    Computing Model Progress CMS Internal Review of Software and Computing

  • Networks, Grids and HENPNext generation 10 Gbps network backbones are almost here: in the US, Europe and Japan First stages arriving in 6-12 monthsMajor International links at 2.5 - 10 Gbps in 0-12 monthsThere are Problems to be addressed in other world regionsRegional, last mile and network bottlenecks and quality are all on the critical pathHigh (reliable) Grid performance across network means End-to-end monitoring (including s/d host software) Getting high performance toolkits in users hands Working with Internet E2E, the HENP WG and DataTAG to get this doneiVDGL as an Inter-Regional Effort, with a GGOC Among the first to face and address these issues

    Computing Model Progress CMS Internal Review of Software and Computing

  • Agent-Based Distributed System: JINI Prototype (Caltech/NUST)

    Includes Station Servers (static) that host mobile Dynamic ServicesServers are interconnected dynamically to form a fabric in which mobile agents can travel with a payload of physics analysis tasksPrototype is highly flexible and robust against network outagesAmenable to deployment on leading edge and future portable devices (WAP, iAppliances, etc.)The system for the travelling physicistStudies with this prototype use the MONARC Simulator, and build on the SONN study See http://home.cern.ch/clegrand/lia/

    Computing Model Progress CMS Internal Review of Software and Computing

  • 6800 Hosts; 36 (7 I2) Reflectors Users In 56 CountriesAnnual Growth 250%

    Computing Model Progress CMS Internal Review of Software and Computing

    Chart reflects data collected at ARPA, NSF data and this analysis, and speculates on growth in the futureLooking to the future, we believe growth will eventually have to flatten out, because most countries will not spend more on telecommunications in the future than they do todayIf traffic growth continues at the 4x rate, growth will slow near 2008-2010 and will remain at the same rate as the % GDP as voice trafficIf growth slows or accelerates, the i...

Recommended

View more >