GENI Science Shakedown

  • View
    38

  • Download
    0

Embed Size (px)

DESCRIPTION

GENI Science Shakedown. Paul Ruth RENCI / UNC Chapel Hill pruth@renci.org. Motivation. Motivation. Are the current GENI technologies capable of effectively supporting the computing needs of domain science? . Motivation. - PowerPoint PPT Presentation

Text of GENI Science Shakedown

Slide 1

GENI Science Shakedown

Paul RuthRENCI / UNC Chapel Hillpruth@renci.org

Background: I work on the ExoGENI testbedRENCI has domain scientists who are interested in ExoGENI (and GENI in general)1MotivationThe 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYWe want to answer the following questions..2MotivationAre the current GENI technologies capable of effectively supporting the computing needs of domain science? The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYMotivationAre the current GENI technologies capable of effectively supporting the computing needs of domain science? What are the benefits and limitations of using GENI for domain science?The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYThis started about a year ago when I was running high-throughput workflows on ExoGENI and found that network wasnt the bottleneck.

We think that I/O will be an issue. Also, performance isolation. But our real expectation is that the bottlenecks will be something we do not yet expect.

4ObservationsThe 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYObservationsGENI has matured to the point where it is now time to evaluate its capabilities with respect to domain science applications at scale and use the results of this evaluation to inform the direction of future GENI development.The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYObservationsGENI has matured to the point where it is now time to evaluate its capabilities with respect to domain science applications at scale and use the results of this evaluation to inform the direction of future GENI development.There is an opportunity to introduce domain scientists to the GENI-way-of-thinking about networking and distributed systems in order to seed the dissemination of GENI technologies to existing and future high throughput and high-performance computing centersThe 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYPersonnelComputer Scientists:Paul Ruth, pruth@renci.orgAnirban Mandal, anirban@renci.orgDomain Scientists:Jeffery L. Tilson, jtilson@renci.orgBrian Blanton, bblanton@renci.org

The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYPeople we brought together to answer these questions

Jeff: Computational Chemist (and sometimes Biologist)Brian: Oceanographer studying storm surge8Project GoalsApply the GENI experiment workflow to domain science applications and evaluate GENIs ability to run domain science experiments PerformanceEase-of-use.Provide feedback to GENI rack developers on the current capabilities with respect to science applications as well as target areas for improvement.Build a tool for domain scientists to create RSpec from high-level descriptions of high-throughput and high-performance applications.

The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYProject GoalsApply the GENI experiment workflow to domain science applications and evaluate GENIs ability to run domain science experiments (performance and ease-of-use).Build a tool for domain scientists to create RSpec from high-level descriptions of high-throughput and high-performance applications.Provide feedback to GENI rack developers on the current capabilities with respect to science applications as well as target areas for improvement.

Bonus Goal: Attract domain scientists to GENIThe 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYI only say its a bonus goal because it is difficult to state a concrete deliverable.10Domain Science ApplicationsMotifNetworkJeffery L. Tilson (RENCI)High Throughput Computational GenomicsIdentification of functional domains and the identification of conserved functional relationships across large numbers of genomes.Workflow ensemble (Pegasus/HTCondor)Task data sets on the order of 50-100 GB.Implicit iterationScales to 1000s of tasks

The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYDomain Science ApplicationsADCIRCBrian Blanton (RENCI)Storm surge and tide modelFinite element modelMPI tightly coupledUrgent ComputingApproved by FEMA for computing storm surge flood hazard simulationsUsed for Digital Flood Insurance Rate Maps (DFIRMs)Scales to 10000+ MPI processes

The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYExperimentsProvisioning overheadcreating, destroying, and modifying slices.GENI for big-data. Large workflow inputs/outputGENI for MPI. Tightly coupled

The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYJust a starting point. We really dont know what we are going to find.

13Outreach to Domain ScienceTool for deploying domain science applications easily.RSpec generation of a Pegasus/HTCondor cluster and MPI cluster.Extendable to other science slice types.Introducing domain scientists to GENI.Feedback from domain scientists to inform future GENI development

The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYFinal thoughts on what we will findMost problems will be on the edgesDisk I/OLocal diskNetwork storageNetwork virtualization mechanismTradeoffs of: virtio, OVS, linux bridges, SR-IOVMore overhead than we realizePerformance isolationSomething unexpected15The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYFirst year goalsGEC 19Applications working on InstaGENI and ExoGENIInitial performance profileGEC 20Deeper performance profileInitial suggestions for rack developers16The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NYThanksQuestions?17The 18th GENI Engineering Conference October 27-29, 2013 Polytechnic Institute of New York University, Brooklyn, NY