Interoperable Mesh Tools for Petascale Applications

Lori Diachin, LLNLRepresenting the ITAPS Team

2

ITAPS focuses on interoperable meshing and geometry services for SciDAC

• ITAPS Goal– Improve SciDAC applications’ ability to take advantage

of state-of-the-art meshing and geometry tools

– Develop the next generation of meshing and geometry tools for petascale computing

• Technology Focus Areas– Complex geometry

– High quality meshes and adaptivity

– Coupled phenomenon

– Dynamic mesh calculations

– Tera/Petascale computing

3

Accomplishing the ITAPS interoperability goal requires a strong team with diverse expertise

Lori DiachinLLNL

Ed d’AzevedoORNL

Jim GlimmBNL/SUNY SB

Ahmed KhamaysehORNL

Bill HenshawLLNL

Pat KnuppSNL

Xiaolin LiSUNY SB

Roman SamulyakBNL

Ken JansenRPI

Mark ShephardRPI

Harold TreasePNNL

Tim TautgesANL

Carl Ollivier-Gooch

UBC

Our senior personnel are experts in complexgeometry tools, mesh

generation, mesh quality improvement,

front tracking, partitioning, mesh refinement, PDE

solvers, and working with application

scientists

Our senior personnel are experts in complexgeometry tools, mesh

generation, mesh quality improvement,

front tracking, partitioning, mesh refinement, PDE

solvers, and working with application

scientists

Karen DevineSNL

4

ITAPS will enable use of interoperable and interchangeable mesh and geometry tools

• Build on successes with SciDAC-1 applications and explore new opportunities with SciDAC-2 application teams

• Develop and deploy key mesh, geometry and field manipulation component services needed for petascale computing applications

• Develop advanced functionality integrated services to support SciDAC application needs– Combine component services together

– Unify tools with common interfaces to enable interoperability

CommonInterfaces

ComponentTools

PetascaleIntegrated

Tools

Build on

Are unifiedby

5

We have a suite of ITAPS services that are built on the ITAPS common interfaces

Mesh Geometry Relations FieldCommonInterfaces

ComponentTools

Are unified by

PetascaleIntegrated

Tools

Build on

MeshImprove

Front tracking

Mesh Adapt

InterpolationKernels

Swapping DynamicServices

Geom/MeshServices

AMRFront tracking

ShapeOptimization

SolutionAdaptive

Loop

SolutionTransfer

PetascaleMesh

Generation

NN

NN

NN

PP

PP

PP PP PP

6

ITAPS technologies impact SciDAC applications in three ways

1. Direct use of ITAPS technology in applications– Geometry tools, mesh generation and optimization for

accelerators and fusion– Mesh adaptivity for accelerators and fusion– Front tracking for astrophysics and groundwater– Partitioning techniques for accelerators and fusion

2. Technology advancement through demonstration and insertion of key new technology areas

– Design optimization loop for accelerators (w/ TOPS)– Petascale mesh generation for accelerators

3. Enabling future applications with ITAPS services and interfaces

– Parallel mesh-to-mesh transfer for multi-scale, multi-physics applications

– Dynamic mesh services for adaptive computationsCurrent

ITAPS/TOPS/SLAC shape optimization

activities will improve ILC design process

7

Work with the COMPASS project spans many different ITAPS service areas

• Mesh Control– Correct curvilinear meshes that properly satisfy

the geometric approximation requirements

• Adaptive Mesh Refinement• Parallel Mesh Generation• Embedded Boundary Discretization• Partitioning Techniques• Geometry Search and Sort

ILC cryomodule consisting of 8 TDR cavities

Beamline Absorber

8

Mesh correction for curves required new technology developments

• Bezier higher order mesh shape– Analytical validity determination– Determine key mesh entity causing invalidity

• Curved mesh modifications for invalid elements– Reshape, split, collapse, swap and refinement

Curved Edge Split to Fix Model Tangency

Curved Region Split

Reshape

9

Results allowed simulations to run faster and further than before

• Initial mesh– 108k mesh regions– 250 invalid regions– Solution blows-up unless

negative contribution removed

• Corrected mesh– No invalid regions– Solution process and 37.8%

faster (CG iterations per time step reduced)

Valid curved mesh after operations

2.97M regions by correcting 1,357 invalid curved regions

10

Goals for curved meshes in thin sections

– Thermal/Mechanical multiphysics simulations– Curved anisotropic meshes for thin sections for

computational efficiency

11

Technology Developments

• Automatically identify thin sections for complex geometry

• Construct curved anisotropic layer elements with proper order

12

Initial results are promising; working with SLAC on use in simulation setting

Straight-sided and curved anisotropic mesh for the cell

model

Close-up of the three curved layer

elements on top of the model faces

2,664 tetrahedral regions

To Do: Technically complex problem; may need fine-tuning in application setting

13

New area: use moving mesh refinement regions to increase computational efficiency

• SciDAC codes - Pic3P/Pic2P– Isotropic refinement for

defined particle domains– Coarse possible meshes

for the rest domains– Pre-defined particle domain

is time-dependent– Achieve computational efficiency

and accuracy

• Technical development– Built on the mesh modification tool

(does all the mesh level work)– Define moving mesh size field with blend

• PIC in long structures

14

Early results show significant savings

New Procedure using Adaptive Controlled Mesh

- 1,030,121 Regions

Compared to a Uniform Mesh -

4,880,593 Regions

To Do:

• Iterate to determine best use

• Correct small number of poor quality elements

15

Adapted mesh (23,082,517 tets)Initial mesh

(1,595 tets)

Parallel adaptive loop for SLAC accelerator design

– Geometry defined in CAD modeler– Omega3P code from SLAC– High level modeling accuracy needed– Adaptive mesh control to provide

accuracy needed– Adaptive loop runs in serial and

parallel

To Do:

• Initial results used file transfer; need to coordinate parallel I/O

• Ensure results are satisfactory

16

Parallel adaptive loop for SLAC uses many different software components

Using geometry operatorsmeans alternate solid modelers can be inserted

Using geometry operatorsmeans alternate solid modelers can be inserted

Using ITAPS mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted

Using ITAPS mesh operatorsmeans alternate mesh generatorsand mesh adaptation procedurescan be inserted

Using ITAPS field operators allows easy construction of alternative error estimators

Projection-based error estimatorused to construct new mesh sizefield given to mesh modification

Mesh adaptation based on local modification linkeddirectly to CAD

Mesh adaptation based on local modification linkeddirectly to CAD

UnalteredSLAC code

UnalteredSLAC code

Error estimatorsfrom RPI and SLAC

Error estimatorsfrom RPI and SLAC

17

• Optimizing a cavity design is still mostly a manual process• Future accelerators employ complex cavity shapes that require

optimization to improve performance

• Geometry & meshing support:

blal a1 a2

b1

arbr

b

ra1ra2

zcl zcrzcbzcc

zcll

Fixed mesh topology:ConvergenceNo re-meshingRe-use factorization

Shape Optimization for Accelerator Cavity Design

Omega3PSensitivity

optimizationITAPS

Geometry & Mesh Update Services

Omega3P

18

GeometryGeometry

Mesh Mesh UpdateUpdate

ITAPS Geometry & Mesh Update Services

ProjectionProjectionxr(d’)→ G(d’)

δd, d’

xr(d)xΩ(d)

xr(d’)

Omega3P

d : Design parameter vector d’ : d + δd, new design parameters

G(d)↔xr(d) : Geometry-mesh classification G(d’) : Geometry for parameters d’

xr(d’), xΩ(d’) : Mesh for new iteration

G(d’)

Design VelocityDesign Velocity∂xr / ∂G, ∂n/∂xr

∂G/ ∂dn(xr(d’))

Omega3PSensitivity

xr(d) xr(d’)xΩ(d’)

ddx ,

19

Services provided by ITAPS DDRIV tool

• Parameterized geometric model construction

– You write function which constructs model using iGeom

– DDRIV acts as driver and handles IO

• Coordination of mesh smoothing on geometric model

• Re-classification of “old” mesh on “new” model

• Target matrix-based smoothing of re-classified mesh

• Computation of design velocities & embedding on mesh using iMesh Tags

Smooth Curves Smooth VolumeSmooth SurfacesNew geom, old mesh Project to CAD,

inverted elements

To Do:

• Incorporate with TOPS/SLAC optimization tools

• Parallelize all aspects

20

Parallel Mesh Generation Needed for Petascale Simulations

• Modeling long range electro-magnetic effects in the ILC requires meshes 4X – 8X current meshes

• Status: – Have a prototype CAD-based parallel mesh

generation capability– Demonstrated good scaling for SLAC NLC

accelerator

• To Do: update to production capability– Adaptive size control– Link to mesh partitioning/load balancing

21

Dynamic load balancing and partitioning via the Zoltan toolkit

• Reduce total execution time – Distributing work evenly among processors– Reducing applications’ interprocessor

communication– Keeping data movement costs low

• Important in many SciDAC technologies– Adaptive mesh refinement– Particle-in-cell methods– Parallel remeshing– Linear solvers and preconditioners

Adaptive Mesh Refinement

Particle Simulations

22

Contact Information

ITAPS• Web Site: www.itaps-scidac.org• Email: itaps-mgmt@lists.llnl.gov• Lori Diachin: diachin2@llnl.gov

We actively seek and welcome your input!

23

LLNL Disclaimer and Auspices

This document was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor the University of California nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or the University of California, and shall not be used for advertising or product endorsement purposes.

This work was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48.