ScienceNetworks:HowDataGetsThere

EliDart,NetworkEngineerESnetScienceEngagementLawrenceBerkeleyNa=onalLaboratory

ATPESC2017

Chicago,IL

August4,2017

Outline

8/4/172

•  ScienceNetworks–structureandrela=onshiptotherestoftheInternet

•  DatatransferatHPCfacili=es

•  Dataportals–past,present,andfuture

NCARRDADataPortal

•  Let’ssayIhaveanicecomputealloca=onattheALCF–climatescience

•  Let’ssayIneedsomedatafromNCARformyproject

•  hSps://rda.ucar.edu/

•  Datasets(therearemanymore,butthesearetwo):

•  hSps://rda.ucar.edu/datasets/ds199.1/(1.5TB)•  hSps://rda.ucar.edu/datasets/ds313.0/(430GB)

•  DownloadtoALCF(couldalsodoNCSAorNERSCorOLCF)

8/4/173

WhatIsAScienceNetwork?•  Downloadingdatafromaportalhappensviathenetwork•  Whatdoes“viathenetwork”actuallymean?•  Whatis“thenetwork”anyway?

•  Mostofusarefamiliarwiththeno=onofanISP–  Internetaccessathome(Ne`lix,etc.)–  Dataforphones(Facebook,maps,Google,etc.)–  Thisis“theInternet”thatmostpeoplesee

•  Sciencenetworksinterconnectscien=ficsites–  HPCfacili=es–  Par=cleaccelerators(LHC,lightsources,…)–  Dataportals

•  SciencenetworksusethesameprotocolsastherestoftheInternet–  TheyarealsoconnectedtotherestoftheInternet

8/4/174

ThisisnotanISP.

It’saDOEuserfacilityengineeredandopDmizedforBigDataScience

Wedothisbyofferinguniquecapabili=esandop=mizingthefacilityfordataacquisi=on,data

placement,datasharing,datamobility.

TheInternet

•  TheInternetiscomposedofalargenumberofindividualnetworks–  Eachisrunbysomeen=tyforitsownreasons

•  Google•  USDepartmentofDefense•  FordMotorCompany•  USDepartmentofEnergy•  AT&T

–  Eachnetworkconnectstoothersforitsownreasons

•  Ingeneral,networksaremorevaluablewhenconnectedtoeachother–  Butremember–thisconnec=vityhappensforselfishreasons–  Notallnetworksarethesame–eachexistsforitsownreasons

8/4/177

Selectednetworksandtheirmissions

8/4/178

Google(Search, YouTube, Gmail, Gdocs, etc.)

AT&T(Phones, home broadband, etc.)

ESnet(Big Science facilities,

DOE labs)

Insurance Company(Business

operations)

ISP(Internet connectivity for

customers, related services)

Internet2(Science network

connectivity for state and regional science

networks)

GEANT(Network connectivity for

European national science networks)

Regional Networks(Network connectivity

for universities, libraries, schools)

European NRENs(Network connectivity

for universities, libraries, schools)

Millions of phones, millions of homes

Giant data centers

National Labsand Facilities

Universities, schools, libraries

Universities, schools, libraries Many of these!

Many of these!

Car Company(Business

operations)

Many of these!

Notesaboutdifferentnetworks

•  Thepreviousdiagramisadras=csimplifica=on–  hSp://www.caida.org/research/topology/as_core_network/2015/

•  Keypoints:–  Allnetworksexistforaspecificreason

•  Somenetworksprovideconnec=vitybetweennetworks•  Somenetworksprimarilyservetheirownusers•  Somenetworksprovideservicestouserswhoaccessthemviadifferentnetworks(e.g.

Google)–  Theselinesareblurry,butit’sausefulwaytothinkaboutit

•  Networkmissioninfluencesengineering,policy,reliability,etc.–  Notallnetworksarebuiltthesameway–  Notallnetworkscansupportallusemodels–  Sciencenetworkshaveadifferenttrafficprofilethancommercialnetworks

8/4/179

ElephantDatavs.MiceData

10

ElephantDatavs.MiceDataBehavior

11

Physicalpipethatleakswateratrateof.0046%byvolume.

è è

Network‘pipe’thatdropspacketsatrateof.0046%.è è

Result100%ofdatatransferred,slowly,at<<5%op=malspeed.

ElephantFlowsPlaceGreatDemandsonNetworks

Result99.9954%ofwatertransferred,at“linerate.”

essen=allyfixed

determinedbyspeedoflight

Throughcarefulengineering,wecanminimizepacketloss.

ElephantflowsrequireessenDallylosslessnetworks

MetroArea

Local(LAN)

Regional

Con=nental

Interna=onal

Measured (TCP Reno) Measured (HTCP) Theoretical (TCP Reno) Measured (no loss)

13

.SeeEliDart,LaurenRotman,BrianTierney,MaryHester,andJasonZurawski.TheScienceDMZ:ANetworkDesignPaSernforData-

IntensiveScience.InProceedingsoftheIEEE/ACMAnnualSuperCompu=ngConference(SC13),DenverCO,2013.

EmergingglobalconsensusaroundScienceDMZarchitecture.

•  Over 120 universi=es in the US have

deployedthisESnetarchitecture.

•  NSF has invested >>$80M to accelerateadop=on.

•  Australian, Canadian, Bri=sh, Brazilianuniversi=esfollowingsuit.

•  hQp://fasterdata.es.net/science-dmz/

1.  Fric=on-freenetworkpath

2.  Dedicateddatatransfernodes(DTNs)

3.  Performancemonitoring(perfSONAR)

ThePetascaleDTNProject•  BuiltontopoftheScienceDMZmodel•  EfforttoimprovedatatransferperformancebetweentheDOEASCRHPCfacili=esatANL,LBNL,andORNL,andalsoNCSA.–  Mul=plecurrentandfuturescienceprojectsneedtotransferdatabetweenHPCfacili=es

–  Performancegoalis15gigabitspersecond(equivalentto1PB/week)–  Realizeperformancegoalforrou=neGlobustransferswithoutspecialtuning

•  Referencedatasetis4.4TBofcosmologysimula=ondata

8/4/1715

DTNClusterPerformance–HPCFaciliDes

16 – ESnet Science Engagement (engage@es.net) - 8/4/17 ©2015,EnergySciencesNetwork

11.8 Gbps

20.2 Gbps

15.2 Gbps

15.1 Gbps

20.6 Gbps 19.7 Gbps

23.0 Gbps

25.7 Gbps

27.2 Gbps

22.9 Gbps

19.4 Gbps

21.2 Gbps

DTN

DTN

DTN

DTN

alcf#dtn_miraALCF

nersc#dtnNERSC

olcf#dtn_atlasOLCF

ncsa#BlueWatersNCSA

Data set: L380Files: 19260Directories: 211Other files: 0Total bytes: 4442781786482 (4.4T bytes)Smallest file: 0 bytes (0 bytes)Largest file: 11313896248 bytes (11G bytes)Size distribution:

1 - 10 bytes: 7 files10 - 100 bytes: 1 files100 - 1K bytes: 59 files1K - 10K bytes: 3170 files10K - 100K bytes: 1560 files100K - 1M bytes: 2817 files1M - 10M bytes: 3901 files10M - 100M bytes: 3800 files100M - 1G bytes: 2295 files1G - 10G bytes: 1647 files10G - 100G bytes: 3 files

June 2017L380 Data Set

ScienceDataPortals

•  Largerepositoriesofscien=ficdata–  Climatedata–  Skysurveys(astronomy,cosmology)–  Manyothers–  Datasearch,browsing,access

•  Manyscien=ficdataportalsweredesigned15+yearsago–  Single-web-serverdesign–  Databrowse/search,dataaccess,userawarenessallinasinglesystem–  Allthedatagoesthroughtheportalserver

•  Inmanycasesbydesign•  E.g.embargobeforepublica=on(enforceaccesscontrol)

8/4/1717

LegacyPortalDesign

10GE

Border Router

WAN

Firewall

Enterprise

perfSONAR

perfSONAR

Filesystem(data store)

10GE

Portal Server

Browsing pathQuery pathData path

Portal server applications:· web server· search· database· authentication· data service

8/4/1718

•  Verydifficulttoimproveperformancewithoutarchitecturalchange–  Soxwarecomponentsalltangledtogether

–  DifficulttoputthewholeportalinaScienceDMZbecauseofsecurity

–  EvenifyoucouldputitinaDMZ,manycomponentsaren’tscalable

•  Whatdoesarchitecturalchangemean?

ExampleofArchitecturalChange–CDN

•  Let’slookatwhatContentDeliveryNetworksdidforwebapplica=ons

•  CDNsareawell-deployeddesignpaSern(e.g.AirBnB,OlympicGames,etc.)

• WhatdoesaCDNdo?–  Storesta=ccontentinaseparateloca=onfromdynamiccontent•  Complexityisn’tinthesta=ccontent–it’sintheapplica=ondynamics•  Webapplica=onsarecomplex,full-featured,andslow•  Dataserviceforsta=ccontentissimple–justmovethefile

–  Separa=onofapplica=onanddataserviceallowseachtobeop=mized

8/4/1719

ClassicalWebServerModel

8/4/1720

•  Webbrowserfetchespagesfromwebserver–  Allcontentstoredonthewebserver–  Webapplica=onsrunonthewebserver–  Webserversendsdatatoclientbrowseroverthenetwork

•  Perceivedclientperformancechangeswithnetworkcondi=ons–  Severalproblemsinthegeneralcase–  Latencyincreases=metopagerender–  Packetloss+latencycauseproblemsforlargesta=cobjects

HostingProvider

TransitNetwork

Residential BroadbandWEB

Long Distance / High Latency

Web Server

Browser

SoluDon:PlaceLargeStaDcObjectsNearClient

HostingProvider

TransitNetwork

Residential BroadbandWEB

Long Distance / High Latency

CDN

DATA

Short Distance / Low Latency

Web Server

CDN Data Server

Browser

8/4/1721

•  CDNprovidessta=ccontent“close”toclient•  Webservers=llmanagescomplexbehavior

•  Latencygoesdown–  Timetopagerendergoesdown–  Sta=ccontentperformancegoesup

•  Loadonwebservergoesdown(noneedtoservesta=ccontent)

•  Significantwinforwebapplica=onperformance

ArchitecturalExaminaDonofDataPortals

•  Commondataportalfunc=ons(mostportalshavethese)–  Search/query/discovery–  Datadownloadmethodfordataaccess–  GUIforbrowsingbyhumans–  APIformachineaccess–ideallyincorporatessearch/query+download

•  Performancepainisprimarilyinthedatahandlingpiece–  Rapidincreaseindatascaleeclipsedlegacysoxwarestackcapabili=es–  Portalserversoxenstuckinenterprisenetwork

•  Canwe“disassemble”theportalandputthepiecesbacktogetherbeSer?–  UseScienceDMZasapla`ormforthedatapiece–  AvoidplacingcomplexsoxwareintheScienceDMZ

8/4/1722

LegacyPortalDesign

10GE

Border Router

WAN

Firewall

Enterprise

perfSONAR

perfSONAR

Filesystem(data store)

10GE

Portal Server

Browsing pathQuery pathData path

Portal server applications:· web server· search· database· authentication· data service

8/4/1723

Next-GeneraDonPortalLeveragesScienceDMZ

10GE10GE

10GE

10GE

Border Router

WAN

Science DMZSwitch/Router

Firewall

Enterprise

perfSONAR

perfSONAR

10GE

10GE

10GE10GE

DTN

DTN

API DTNs(data access governed

by portal)

DTN

DTN

perfSONAR

Filesystem (data store)

10GE

Portal Server

Browsing pathQuery path

Portal server applications:· web server· search· database· authentication

Data Path

Data Transfer Path

Portal Query/Browse Path

8/4/1724

PutTheDataOnDedicatedInfrastructure

•  Wehaveseparatedthedatahandlingfromtheportallogic•  Portaliss=llitsnormalself,butenhanced

–  PortalGUI,database,search,etc.allfunc=onastheydidbefore–  QueryreturnspointerstodataobjectsintheScienceDMZ–  Portalisnowfreedfrom=estothedataservers(runitonAmazonifyouwant!)

•  Datahandlingisseparate,andscalable–  High-performanceDTNsintheScienceDMZ–  Scaleasmuchasyouneedtowithoutmodifyingtheportalsoxware

•  Outsourcedatahandlingtocompu=ngcenters–  Compu=ngcentersaresetupforlarge-scaledata–  Letthemhandlethelarge-scaledata,andlettheportaldotheorchestra=onofdataplacement

8/4/1725

DataPortalImplicaDons

•  Portalsholdalotofvaluabledata–  Observa=ons(skysurveys,satellitedata,genomes,etc.)–  Manyhavebeeninplaceforyears

•  Mostareinadequatetosupportlarge-scaleanalysis–  Legacysearch/queryinterfaces–  Legacyaccessprotocols/tools–  Thisisintheprocessofchanging

•  Thetechnologyexiststoradicallyimprovetheu=lityofdataportals–  Whatshouldtheperformanceexpecta=onbe?–  HPCfacili=escando1PB/week–ifdataportalscoulddothis…

8/4/1726

NCARRDAPerformancetoDOEHPCFaciliDes

13.9 Gbps 16.6 Gbps 11.9 Gbps

DTN

nersc#dtnNERSC

DTN

olcf#dtn_atlasOLCF

DTN

alcf#dtn_miraALCF

DTN

NCAR RDArda#datashare

8/4/1727

•  1.5TBdataset

•  1121files

Summary

•  Sciencenetworksareengineeredtosupportdata-intensivescience–  RelatedtoandconnectedtotherestoftheInternet,butdifferent

•  ScienceDMZmodeleffec=velyconnectsdatainfrastructuretonetworks–  Ifyouneedtosendyoursysadmintome,feelfree

•  GlobusatHPCfacili=esmakesterascaletopetascaledatatransferspossible–  (moreonGlobuslatertoday)

•  HugeopportunityinupgradingdataportalstouseScienceDMZ,DTNs,advancedtools(e.g.Globus)–  MakelargedatarepositoriesavailableforanalysisatHPCfacili=es

8/4/1728

Inconclusion–ESnet’svision:

Scien=ficprogresswillbecompletelyunconstrainedbythephysicalloca=onofinstruments,people,computa=onal

resources,ordata.

29

LinksandLists

–  ESnetfasterdataknowledgebase•  hSp://fasterdata.es.net/

–  ScienceDMZpaper•  hSp://www.es.net/assets/pubs_presos/sc13sciDMZ-final.pdf

–  ScienceDMZemaillist•  Sendmailtosympa@lists.lbl.govwithsubject"subscribeesnet-sciencedmz”

–  perfSONAR•  hSp://fasterdata.es.net/performance-tes=ng/perfsonar/•  hSp://www.perfsonar.net

–  Globus•  hSps://www.globus.org/

30 – ESnet Science Engagement (engage@es.net) - 8/4/17 ©2015,EnergySciencesNetwork

Thanks!

EliDartEnergySciencesNetwork(ESnet)LawrenceBerkeleyNa=onalLaboratory

hSp://my.es.net/

hSp://www.es.net/

hSp://fasterdata.es.net/