Upload
alisha-hardy
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
Enabling HPC Simulation Workflows for Complex Industrial Flow Problems
C.W. Smith, S. Tran, O. Sahni, and M.S. Shephard, Rensselaer Polytechnic Institute
S. SinghIndiana University
Outline Industry requires complete HPC workflowsRPI efforts on HPC for industryComponents for parallel adaptive simulation Science GatewayApplication to complex industrial flow problems
HPC for Industry
Increasingly Industry requires parallel analysis to meet their simulation needswith key drivers being Higher spatial and temporal resolution More complex physics with many multiphysics problems Increased use of validation and movement toward
uncertainty quantification Reasonable progress being made on the analysis engines
Research codes that scale to nearly 1,000,000 cores on unstructured meshes
Commercial codes improving scaling to thousands for flow More reasonable software pricing models
However, the application of HPC in industry is growing slowly Economics of product design cycle indicate it should be
growing quickly
HPC for Industry
Why is the use of industrial HPC growing slowly? The analysis codes are available. What is missing? To obtain the potential cost benefits the entire simulation
workflow must be integrated into the HPC environment Workflow must included tools industry has spend years
integrating and validating in their processes Need to use multiple CAD and CAE tools
Effective industrial use of large scale parallel computations will demand simulation reliability
Must have very high degree of automation – human in the loop kills scalability and performance
Need easy access to cost effective parallel computersMust be able to do proprietary work Must have easy to use parallel simulation management
HPC for Industry
Approach being taken A component-based approach to integrate from design
through results quantificationLink to industry design data (e.g., CAD geometry)Manage the model construction directly on massively
parallel computersSupport the use of multiple analysis engines Support simulation automation
Support in-memory integration of components as much as possible to avoid I/O bottlenecks
Provide web-based portal for execution of massively parallel simulation workflows
This presentation will focus on components developed for parallel adaptive unstructured mesh simulations
Rensselaer’s Efforts to Bringing HPC to Industry
Scientific Computation Research Center (SCOREC) Parallel methods for unstructured
meshes and adaptive simulation controlComponent-based methods for
developing parallel simulation workflows Center for Computational Innovations
Petaflop IBM Blue Gene/Q and clustersIndustry can gain guaranteed access
to run proprietary applications (for a price less that cloud computing)
Programs for HPC for IndustryHPC2 – New York State HPC consortiumNSF Partnership for InnovationNSF XSEDE Industrial Challenge Program
5
iM0
jM1
1P
0P 2P
off-node part boundary
on-node part boundary
Node j
Node i
SCOREC’s Research Builds on Broad Partnerships
Interdisciplinary research program supported by Government – NSF, DOE, DoD, NASA, NIH, NY State Strong industrial support – 46 different companies have
supported SCOREC Multiple pieces of software have been commercialized Center has generated a software vendor Multi-way partnerships are common
Large industry, software vendor, SCORECSBIR from government agencies to software
vendor and SCOREC Government laboratory,
software vendor(s), SCOREC
University, SCOREC, etc.
Center for Computational Innovations
IBM Blue Gene/Q petaflop computer• 5120 compute nodes- 5 racks @ 1024 nodes each
• Each node has 16 A2 processing cores- 17th core for OS functions
• 16 GB of RAM per node• 80 TB of RAM system wide• 56 Gb/s IB external network• 160 nodes for data I/O• 1.2 PB parallel file system
High Performance Computation Consortium
HPC2 supported by the NYSTAR Division of the Empire State Development Agency
Goal is to provide NY State Industry support in the application of high performance computing technologies in:Research and discoveryProduct developmentImproved engineering and manufacturing processes
HPC2 works with NY State Centers for Advanced Technology which serve as focal points for technology transfer to Industry
The HPC2 is a distributed activity - key participantsRensselaerStony Brook/Brookhaven SUNY BuffaloNYSERNET
8
x/h=-6z/h=16
x/h=-6z/h=-16
x/h=25z/h=16
x/h=-6z/h=16
x/h=-6z/h=-16
x/h=25z/h=16
Time-avg(Cb=1.2)
Exp CFD
Simulation of active flow control device (Sahn, et al.)
NSF Sponsored Activities on HPC for Industry
Partnership for Interoperable Components for Parallel Engineering Simulations Technologies to make construction of HPC workflows
more efficient Component-based methods supporting combinations of
open source and commercial software Mechanisms to help industry effectively apply HPC
NSF XSEDE Industrial Challenge Program Install components for parallel adaptive simulations on
XSEDE machines Develop HPC workflows for industry on XSEDE machines Investigate use of Phi co-processors on the Stampede
system for in parallel adaptive unstructured mesh simulations
9
Recent Industrial Partners Industrial Partners
ACUSIM (now Altair) Ames-Goldsmith Blasch Ceramics Boeing Calabazas Creek Research Corning Crystal IS GE HyPerComp
IBM ITT Gould Pumps Kitware Northrop Grumman Pliant Procter & Gamble Sikorsky Simmetrix Xerox
Parallel Data & Services Domain Topology
Mesh Topology/Shape
Dynamic Load Balancing
Simulation Fields Partition Control
Component-Based Unstructured Mesh Infrastructure
Parallel data and services are core Abstraction of geometric model
topology (GMI or GeomSim) Mesh also based on topology – it
must be distributed (PUMI or MeshSim), growing need for distributed geometry (GeomSim)
Simulation fields are tensors with distributions over geometric model and mesh entities (APF or FieldSim)
Partition control must coordinate communication and partition updates
Dynamic load balancing required at multiple steps in the workflow to account for mesh changes and application needs (Zoltan and ParMA)
PUMI, GMI, APF, ParMa are SCOREC research codes
GeomSim, MeshSim and FieldSim are component-based tool from Simmetrix
Zoltan is from Sandia Nat. Labs
iM0
jM11P
0P 2P
off-node part boundary
on-node part boundary
Node j Node i
Distributed Mesh and Partition Control
Distributed Mesh Requirement On part operated without communication Communications through partition model
Services Mesh Migration – moving
mesh between parts Ghosting – read only copies
to reduce communication Changing numbers of parts
Geometric model Partition model
iM0
jM0
1P
0P 2P
inter-process boundary
intra-process part boundary
process j process i
Distributed mesh
Dynamic Load Balancing
Equal “work” with minimum communication Tools
Graph-based (ParMETIS, Zoltan) Geometry-based (Zoltan, Zoltan2) Mesh-based (ParMA) Local/global
Load balancing throughout simulation Need fast methods – can not dominate Need predictive load balancing to account
for mesh adaptation Need to account for needs of
specific workflow components
Gateway Execution
High barrier to run HPC workflows Requires knowledge of filesystem,
scheduler, scripting, runtime env., compilers, … - for each HPC system
XSEDE science gateway for PHASTA lowers the barrier User specifies problem definition,
simulation parameters, and required compute resources through experiment creation web page (right)
Workflow steps are executed on HPC system, user is emailed, and output is prepared for download – option to delete or archive
Scales to multiple users and systems
Gateway Creation and Maintenance
System and user software expert maintains builds and execution scripts Optimized builds and runtime parameters Web interface for defining workflow
XSEDE gateway developers quickly accommodate user requests through SciGap and Airavata APIs Output log monitoring – monitor job output from
web interface Email notifications – completion, failure, app
specific milestone, … Data persistence – industrial user wants data
deleted after run Configuring HPC system access – adding RPI
BlueGene /Q supportTwin-screw extruder axial velocity: (left) two threads of the screw and (right) cross-section across the extruder.
PHASTA gateway experiment summary.
Parallel Data & Services
Domain Topology
Mesh Topology/Shape
Dynamic Load Balancing
Simulation Fields
Physics and Model Parameters Input Domain Definition with Attributes
Mesh-Based Analysis
Complete Domain
Definition
Mesh Generation and/or Adaptation
Postprocessing/Visualization
SolutionTransfer
Correction Indicator
PDE’s anddiscretizationmethods
Solution transfer constraints
mesh with fields
mesh with fields
calculated fields
mesh size field
meshes and fields
meshing operation geometric
interrogation
Attributed
topology
non-manifoldmodel construction
geometry updates
mesh size field
mesh
Partition Control
Component-Based Unstructured Mesh Infrastructure
Component-Based Unstructured Mesh Infrastructure
File transfer a serious bottleneck in parallel simulation workflows All core parallel data and services accessed through APIs File-based workflows require no change of components
Often first implementations done via files, but using APIs In-memory integration approaches use APIs
Support effective migration from file-based to in-memory for “file-based” codes – replace I/O routines with routines that use APIs for transfer between data structures
For more component-based codes the in-memory integration was easier to implement than file based
In-memory has far superior parallel performance
Adaptive Loop Applications
Adaptive loops to date have been used for Modeling of nuclear accidents and various flow problems with
University of Colorado’s PHASTA code Solid mechanics applications with Sandia’s Albany code Modeling fusion MHD with PPPL’s M3D-C1 code Accelerator modeling problems with SLAC’s ACE3P code Aerodynamics problems with NASA’s Fun3D code Waterway flow problems with ERDC’s Proteus code High-order fluids simulations with Nektar++
Modeling a dam break
Plastic deformation of a mechanical part
Parallel Data & Services
Domain Topology
Mesh Topology/Shape
Dynamic Load Balancing
Simulation Fields
Physics and Model Parameters Input Domain Definition with Attributes
PHASTA
Parasolid or
GeomSim
MeshSim and MeshSim Adapt
Paraview
SolutionTransfer
Hessian-based error indicator
NS, FELevel set
Solution transfer constraints
mesh with fields
mesh with fields
calculated fields
mesh size field
meshes and fields
meshing operation geometric
interrogation
Attributed topology
non-manifoldmodel construction
geometry updates
mesh size field
mesh
Partition Control
Complex Flow Simulations
Active flow control on vertical tail improves its effectiveness. Massively parallel simulations provide tremendous physical insights. Integrated experimental and numerical investigation at UC Boulder and RPI.
Active Flow Control on Vertical Tail
AFC results in reduction of drag/size of the tail.
20
Petascale simulations
Two-phase modeling using level-sets coupled to structural activation
Adaptive mesh control – reduces mesh required from 20 million elements to 1 million
New ARO project using explicit interface tracking totrack reacting particles
Adaptive Two-Phases Flow
Modeling Ceramic Extrusion
Objectives Develop end-to-end workflow
for modeling ceramic extrusion Tools
SimModeler – mesh generation and problem definition
PHASTA – massively parallel CFD Chef – pre-processing, solution
transfer, and mesh adaptation driver Kitware Paraview – visualization
Status and Plans Non-linear material model and partial-
slip boundary condition into PHASTA. Extended pre-processor to support
partial-slip boundary condition. Created XSEDE web-based gateway
for automated execution of workflow. Planning gateway support for CCI.
Velocity and pressure fields.
Twin screw extruder.
Aerodynamics Simulations
Parallel Data & Services
Domain Topology
Mesh Topology/Shape
Dynamic Load Balancing
Simulation Fields
High speed flow scenarios Parasolid
FUN3D from NASA
Parasolid or
GeomSim
MeshSim and MeshSim Adapt or
MeshAdapt
Paraview
SolutionTransfer
Goal oriented error estimator
NS,Finite volumes
Mass conservation
mesh with fields
mesh with fields
flow fields
mesh size field
meshes and fields
meshing operation geometric
interrogation
attributed
topology
non-manifoldmodel construction
mesh size field
mesh
Partition Control
NASA Trap Wing
Zoom of leading edge of the main wing
Adapted: LEV2
Initial: LEV0
Cp plots, near the tip
Summary
Technologies and tools needed to create effective HPC workflows for industry are available However, it is not a “field of dream” – just building the tools
will not get industry to come use them Need to work with industry to create effective simulation
workflows that address their needs Progress is being made on developing the needed tools and
mechanisms – more progress is needed Requires too much expertise Takes too much time/effort
Even with additional improvement,expect that it will still be a “contact sport” requiring interactions betweencomputational scientists and the engineers that will use the simulations