Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Acommonandsustainablebigdatainfrastructureinsupportofweatherpredictionresearchandeducationinuniversities
Unidata ModelingResearchintheCloudWorkshop,5/31/17
CarlosMaltzahn Background• CurrentResearch
• High-performanceultra-scalestorageanddatamanagement
• End-to-endPerformancemanagementandQoS
• ReproducibleEvaluationofSystems
• NetworkIntermediaries
• OtherResearch• DataManagementGames• InformationRetrieval• CooperationDynamics
2
• AdjunctProfessor,ComputerScience,UCSantaCruz
• Director,UCSCSystemsResearchLaboratory(SRL)
• Director,CenterforResearchinOpenSourceSoftware(CROSS)cross.ucsc.edu
• Director,UCSC/LANLInstituteforScalableScientificDataManagement(ISSDM)
• 1999-2004:PerformanceEngineer,Netapp
• Advising 6Ph.D.students.• Graduated 5 Ph.D.students• Idothis100%ofmytime!
Project• NSF-fundedScientificSoftwareIntegrationproject(SSI)oftheSoftwareInfrastructureforSustainedInnovation(SI2)program• Goal:sustainablecommunitySWframework
Collaborators
CarlosMaltzahn,IvoJimenez(UCSantaCruz),
JoshHacker,JohnExby,KateFossell (NCAR),
MohanRamamurthy(Unidata),
GretchenMullendore,TimothySee(UND),
BrianAncell (TexasTech),
WilliamCapehart (SDSM),
ClarkEvans(UWMilwaukee),
RobertFowell,KevinTyle (UAlbany),
StevenGreybush (PennState),
RussSchumacher(CSU).
ProblemInformedbyEarthCube Usersworkshops• Poorreproducibilityofdata-intensivescience
• Impactoneducationandresearch
• Impairedavailabilityofintermediateresults• Unnecessaryduplicationofwork,steeplearningcurves
• Communitiesofpracticearefallingbehind• Limitedabilitytoadoptnewtechnologies
4
Domain:NumericalWeatherPrediction• NWPgroupsatuniversitiesusesupercomputingtimetocreatelargeensembles• Currentpractice:
• keepensemblesinscratchspaceordownloadtolocalinfrastructure• Don’tshareensembleproducts,don’tsharetools• Rewardsforresults,notdata
5
GeneralApproach
• Establish“nuclei”:piecesoftechnologythat• Areeasilyshareable• Havetheabilitytogrow&improveovertime• Ensure“buy-in”fromresearchersandstudents
• Examples:• Wikipedia• Linuxkernel
• Infrastructurestoenablecommunity-drivenreviewandimprovement
6
BigWeatherWebNuclei
1. Largeensembledistributedover7universities:GretchenMullendore (UND),BrianAncell (TexasTech),WilliamCapehart (SDSM),ClarkEvans(UWMilwaukee),RobertFowell (SUNYAlbany),StevenGreybush (PennState),RussSchumacher(CSU).
2. Commonstorage,linking,andcatalogingmethodology:DataInvestigationandSharingEnvironment• Permanentnamingandhighavailabilityofdataandexperiments• Connectingdata,platform,tools,analysis
3. SoftwareContainertechnologiesforeasydeploymentandreproducibility• Self-contained:softwarecanbeinstantlydeployedincommonenvironments• Namingandversioning:compactreferencemechanismsforcomplexenvironments• Goodforreproducibilityandeducation
7
Nucleus1:Large,distributedensembles
• Testingthedistributedensembleframeworkandtools• Sharingof“knowledgeproducts”
• Initializationmethods• Physicsoptions• Workflowscriptsforproducing&analyzingdata• Success:BWWPis areusingtheBWWensembletodoscience
• Trackingdataauthorshipandcommunityimpact• WehaveaDOIbutaccesshastobemanaged(expenseofdataegress,seebelow)• Ensembleisevolvingovertime
• Disseminationofframework&tools• SeeNCAR’s“WRFinabox”work
Nucleus1:Large,distributedensembles
• Testingthedistributedensembleframeworkandtools• Sharingof“knowledgeproducts”
• Initializationmethods• Physicsoptions• Workflowscriptsforproducing&analyzingdata• Success:BWWPIsareusingtheBWWensembletodoscience
• Trackingdataauthorshipandcommunityimpact• WehaveaDOIbutaccesshastobemanaged(expenseofdataegress,seebelow)• Ensembleisevolvingovertime
• Disseminationofframework&tools• SeeNCAR’s“WRFinabox”work
Nucleus2:Commonstorage,linking,andcatalogingmethodology• Enablefiguresinpublications&teachingmaterialstolinktoenvironments,tools,anddatathatproducedthem• Providedinaformthatisreusable
• Easyinstallofenvironmentandtools• Creationandaccesstodataproductswithoutneedtodownloadeverything• Dataproductsbythemselveslinkbacktotheirantecedentsinareusableway.
Usecloudservicesinsteadofon-premise installations
• Convertshardtechnical,management,andfundingquestionsintojustfundingquestions• Startedwithcommercialcloud:AWS($9k/monthcredit)
• 50TBsofaronS3:$800/month• THREDDSserveronEC2:$160/month• ParticularthankstoJohnExby andKevinTyle
• Challenges:• Commercialcloud:costofdataegress.PlanningmovetoXSEDE/TACC/Wrangler• Long-termmanagementofstoragecommons(withbetter-than-scratch-spacepolicies)
• Long-termnaming:gettingaDOIistheeasypart-- long-termavailability?
11
Nucleus2:Commonstorage,linking,andcatalogingmethodology
Nucleus3:SoftwareContainers
• Seeearliertalk”CollaborativeWRF-basedresearch&education,enabledbysoftwarecontainers”byJoshHacker,JohnExby,andKateFossell• PracticalFalsifiableResearch(Popper,falsifiable.us,seeposter)
• Applyopen-sourcesoftwarecommunitypracticestoexperimentmanagement• Scripteverything,leverageworkflowsystems&DevOpstools
• SeealsoEricKlavins’AquariumProject:klavinslab.org/aquarium.html• Keepeverythingingit repositories• Usesoftwarecontainersforallsoftware
• Namingconventiontoautomaterunningandvalidatingexperiments• Conventionsforcompactcomputingenvironmentdescription• SeeposterbyIvoJimenez
12
BWWOutreach
• 2015Unidata UsersMeeting• 2015AGUTownhall
• ~50attendees• 10newbww-userssubscribers
• 2016PresentationatAMS• 2016Unidata Workshops• WRFinaboxintheclassroom
• 2016UNDclassbyTimSee(UND)• 2017UNDclassbyGretchenMullendore (UND)• Wiliam Capehart (SDSM)usingBWWensembles(2papers)
• Popper• Jimenezetal.VarSys’16,Chicago,IL• Jimenezetal.USENIX;login:,Winter’16• Guestlecturein2017UNDclassbyGretchenMullendore
13
• Websites:• bigweatherweb.org• www.ral.ucar.edu/projects/ncar-docker-wrf• falsifiable.us
• Emaillist:[email protected]• Slack:bigwxweb.slack.com,invitations:KevinTyle,[email protected]• Contact:CarlosMaltzahn,[email protected]