Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things

FACE-IT:EarthscienceworkflowsmadeeasywithGlobusandGalaxytechnologies

(Providemorecapabilityformorepeopleatlowercostbydelivering“ScienceasaService”)

RaffaeleMontella1,2,AlisonBrizius2,JoshuaElliott2,DavidKelly2,RaviMadduri2,3,KetanMaheshwari3,CherylPorter4,PeterVilter2,MichaelWilde2,WeiXiong4,MengZhang4 andIanFoster2,3,5

1DepartmentofScienceandTechnologies,UniversityofNaplesParthenope,Naples,ITALY;2ComputationInstitute,ArgonneNationalLaboratoryandUniversityofChicago,Chicago,Illinois,USA;3MathematicsandComputerScienceDivision,ArgonneNationalLaboratory,Argonne,Illinois,USA;

4UniversityofFlorida,DepartmentofAgriculturalandBiologicalEngineering,Gainsville,Florida,USA;5DepartmetofComputerScience,UniversityofChicago,Chicago,Illinois,USA;

DepartmentofScienceandTechnologiesUniversityofNaplesParthenope

MathematicsandComputerScience

Division

DepartmentofAgriculturaland

Biological Engineering

faceit-portal.org usefaceit.org

FACE-IT:AFrameworktoAdvanceClimate,Economic,andImpactInvestigationswithInformationTechnology

FacingrealproblemswithInformationTechnology

NobuzzwordRealthings!Anopenplaygroundforthenextgenerationofearthsystemscientists

What’sinaname…

ScienceGatew

ays

Data+Workflows=Re

sults

Theuserprofile…Scientists

Expertsoftheir fieldsLimited programmingskills

Complex experiments

Effectiveandefficientsolutions torealproblemsExperts indesign andabstraction

Inform

ation

Techno

logy

Development-experts (inwizardry)…

BuiltonwidelyusedGalaxy,

Globus, andSwift systems

faceit-portal.org

…lessmagicwands,moremagicminds...


ServiceOrientedScience

• Peoplecreateservices(dataorfunctions)…• whichIdiscover(&decidewhethertouse)…• &composetocreateanewfunction...• &thenpublishasanewservice.

→ Ifind“someoneelse”tohost services, soIdon’thavetobecomeanexpertinoperatingservices&computers!→ Ihopethatthis“someoneelse”canmanage security,reliability, scalability,… !!

2005

IaaS

PaaS

SaaS


TheScienceStack

• Galaxy– Interactive execution– Creation,Execution, Sharing,DiscoveringWorkflows

• Globus– Datamanagement– IdentityManagement

• AWS– HTCondor,Chef,EC2,EBS,S3,SNS,NEWT– Spot,Route53,CloudFormation

Makingcomplexthingsstraightforward

Infrastructure

Platform

Software


FromgenestoEarthin6steps

• Datatypes

• Tools

• Toolparameters

TeachingGalaxytospeakEarthScience

• Aggregateddatatypes

• Dataproviders

• Visualizers


StepONEof6:earthsystemdatatypes

• Colorscheme data.Data

data.Text

tabular.Tabular xml.Generic_Xml

enhanced_xml.EnhancedXml generic_json.GenericJson

enhanced_xml.DsRef(RAFT)

geojson.GeoJson geojson.GeometryJsongeojson.BBoxJson geojson.CrsJson

netcdf.NCML

netcdf.NetCDF

ecmwfera40.ECMWFERA40gcm.GCM

acmo.Acmo

RegularGalaxy Extensionsofgeneraldatatypes

Newdatatypesforearthsciences

FACE-ITapplicationsdatatypes

OtherFACE-ITappsdatatypes



StepTWOof6:newtools

• Tool:Isacomputingprocess fedbyoneormoredatasets producing oneormoredatasets

• Itiswrappedoverany kindofexecutable

• Runningbynaïve localscheduler,super-computers,virtualmachinessomewhere inthecloud.

• Eachinputandoutput isdatatyped• Itisdefined usingXML Thetoolspalette

Thesametoolinaworkflow

Atoolindataanalysis

TeachingGalaxytospeakEarthScience(“Globusally”)


StepTHREEof6:toolparameters

• Toolparameters:Definetheuserinterfaceelements foratool

• Regulartoolparameterswraptextfields, radiobuttonsanddropdrownlists.

• Customtoolparameters forGlobus,OpenDap,datepeakingandfeatureselection onmaps.



StepFOURof6:aggregateddatatypes(RAFT*)

• DatasetReferences:XMLbaseddatatypegroupingreferences todifferentdatasets inthesamehistory.

• TheregularGalaxyworksonsingle filedatasets orcomposite filedatasets.

• Actsasa‘struct’oran‘array’oramixofboth.

• Supportsschemas andtranslators.

DsRef(EnhancedXML)

Usedwhen:• Atoolconsumes and/orproducesavariable

numberofdatasets• Thetoolisimplemented usingaSwiftscript

workinginparallel


Globus HTTPS Endpoints(...future…)


StepFIVEof6:dataproviders

• Dataproviders:softwarecomponentsinterfacing thedatasets withthewebbrowser.

• TheyprovidedataasarrayofJSONobjects

• Key/Values, Columnar,custom• Implemented inDatatypeclasses

WebBrowser GalaxyInstance

HistoryDatabaseAssociation

DataProviders

Datatype

DatafileDataset

DataProvider

Webpage…

…dynamicallygenerated...

…formMakotemplate

(mixofserversidepythoncodewithclientsidewebtechnologies)

request

response

template



StepSIXof6(!):mapvisualizers

• Visualizers:client-sidesoftwarecomponentsforinteractivedatavisualization

• Quasi-GIS!

• Map:VisualizesvectordataproducedasGeoJsonobjectsbyadataprovider

• Wms (WorldMapServer):VisualizesrasterdatafromNetCDFdatatypes.



Deliverondemandweathersimulations

• wrf-model.org

• +30kpeoplecommunity

• 150countries

• highcomputingdemanding

WeatherResearchandForecast@AWS:a(real)application.


WeatherForecastonChicagoarea(333m)

• WRF• 4nested

domains• 9kmto333m

Temperature (C°)Cumulated Rain(mm)

WeatherResearchandForecast@AWS


NetCDFscavenging,discoveryandprovisioning

• NetCDFScavenger• Indexing/Ingestion• Tiling• Spatial/metadata

query• Selection• Usage

Breakingthefourthwall

Craweler

Ingester

Tiler

FACE-ITGalaxyToolBox

Discover

Publish

Share

NetCDFdata


Conclusionsand[now]futureworks

• Face-ITGalaxyisacreativeplaygroundforthenextgenerationofearthscientistspoweredbyGlobus fordatamovementandmore.

• Propose yourapplication,writeyourcodeandshareit!

• Spin-offprojects:extremeweathersimulationsintheBayofNapoli,IT(UniParthenope)

http://www.faceit-portal.org

Mustbe...…continued!

Documents

Mathematics and Department of Agricultural and Biological ... · PDF file• AWS – HTCondor, Chef, EC2, EBS, S3, SNS, NEWT – Spot, Route 53, Cloud Formation Making complex things