DATA-CENTRIC COMPUTING, SCIENCE GATEWAYS, AND THE TERAGRID

Lustre-WAN: Enabling Distributed Workflows in Astrophysics Scott Michaelscamicha@indiana.eduApril 2011

03/04/091Studying Planet FormationMy research group investigates the formation of giant planetsThere are more than 500 extra-solar gas giants that have been discovered to dateInvestigations are carried out using a 3D radiative hydrodynamics codeEach simulation takes several weeks and produces 5-10 TB of dataA series of 3-5 simulations is needed for each studyData is analyzed to produce a variety of measurements

03/04/092Our Workflow Before Using Lustre-WANOur workflow has three partsSimulation shared memoryAnalysis distributed memoryVisualization proprietary software with interactivityIn the past we have transferred data between HPC resources, stored the data locally, and performed analysis and visualization

03/04/093Using a Central Lustre File SystemUsing Indiana Universitys Lustre based Data Capacitor WAN file system different systems in separate parts of the workflow can all access the same dataData generated by SGI Altix4 simulations on NCSAs Cobalt6 simulations on PSCs PopleData analyzed by departmental resources and distributed memory machinesIndiana Universitys Big Red and QuarryMississippi State Universitys Raptor and TalonData visualization by departmental machinesIndiana University IDL license03/04/094Using a Central Lustre File System

03/04/095Use Cases for Lustre WANMany cases where researcher needs resources outside a single data centerThis is increasingly common in the TeraGridA user needs to use heterogeneous resourcesShared and Distributed MemoryGPUsVisualization systemsA researcher needs to capture instrument dataA researcher needs to migrate to a different systemIdeally every user can access all his data from any resource all the time03/04/096Thanks ToDC TeamSteve SimmsJosh WalgenbachJustin MillerNathan HealdEric IsaacsonIU Research TechnologiesMatt LinkRobert HenschelTom JohnsonMSU ResearchTrey BreckenridgeRoger SmithJoey JonesVince SandersGreg GrimesPSCNCSA

This material is based upon work supported by the National Science Foundation under Grant No. CNS-052143303/04/097