Jean-François Remacle, Joe E. Flaherty and Mark S. Shephard

Rensselaer Polytechnic Institute remacle@scorec.rpi.edu

Parallel Algorithm Oriented Mesh Datastructure

Outline Basics of Mesh Representation Parallel Extensions Software issues Examples http://www.scorec.rpi.edu/AOMD

AOMD-PAOMD

AOMD and PAOMD deals with meshes A core for basic mesh representation (topological adjacencies)

Advanced design patterns : Iterator, Observer, Visitor... Some extensions

Parallel services : message passing, load balancing, partitioning Meshing Toolbox : quality measures, mesh modifications, cavity mesher,… Calculus : coordinate systems, integration <>

AOMD supports Geometry based analysis design, classification

Parasolid, ProE, STL (not open source yet) Hybrid meshes

Hexes, Tets, Quads … Non conforming meshes

(hanging nodes, AMR) Curved meshes (Bézier)

AOMD is an Open Source Project

Advanced analysis techniques Are automated with automatic mesh generation from geometric models Employ variable order elements based on other than Lagrange basis Use various weak forms that required alternative mesh relationships Adaptive the mesh as the simulation proceeds

To meet these needs the mesh data structure must Understand the relationship of mesh entities with the geometric model -

ensure mesh validity and associate physical parameters with the mesh Understand the interactions of various mesh entities - different

relationships used during automatic mesh generation and analysis Support assignment of independent geometry to the entities in the mesh -

must control geometric approximation with higher order methods Be able to associate dof to various mesh entities - provides flexibility to

support variable p-order and different collections of dofs Effectively maintain relationships during modification - needed in mesh

generation, mesh adaptation and mesh modification for evolving geometry

Motivations

Automated Adaptive Analysis

FEM PUM

AOMD-MeshSim

TrellisAOMD

AOMD-MeshSim

Topological Mesh Data Structure

Classic node point coordinates / element connectivities do not meet this need

Mesh representations based on topological entities and their adjacencies fill the need: Can be proven to be complete and unique -

can effectively support all relationships and associations needed by any mesh generation or analysis procedure

Provide a shape independent abstraction for associating geometry

Effectively supports the linkage to the geometric model since systems maintain a model topology

Various approaches to support meshtopology have been taken

A new approach considered herecurved mesh for

p-version analysis

Modern Geometric Modeling Systems

Employ non-manifold boundary representation

Provide access to model and its geometry through a geometric modeling kernel driven by topological entities

Simulation processes (mesh generation, p-version analysis...) can directly interact with the modeler

Basics of mesh representation

A geometrical domain G Is the highest level representation of the domain is composed of geometrical entities Gi

d

A Mesh M is a discrete representation of G is composed of mesh entities Mi

d, i=1,… Nd(M) together with their

adjacencies Mid {M q }

Mesh entities Mid

4 different topological kinds Vertices (d=0), edges (d=1), faces (d=2) and regions (d=3)

The unique association of a mesh entity, Midi, to a geometric model

entity, Gjdj, where di<dj is denoted by

Midi Gj

dj

Adjacencies sets

Downward Adjacenties

1100

1110

0111

0011

I Upward Adjacenties

One-Level Circular

1100

0110

0011

1001

I

0

1

2

3

M

M

M

M

3M 2M 1M 0M

Face to Edge

Examples of complete adjacencies sets

1100

0110

0011

1001

I

1100

0110

0011

0001

I

Incomplete Complete

Cost of a mesh representation

Memory cost of some mesh representation

Higher order adjacencies

First order adjacencies Direct access, unit cost Full representation : only way to have all adjacencies as first order

Higher order adjacencies sets Regions know faces : Mi

3 {M 2 }

Faces know edges : Mi2 {M 1 }

Edges know vertices : Mi2 {M 1 }

Third order adjacencies sets : Mi3 {M 2 } {M 1 } {M 0 }

Functionally complete representation

Minimum information Equally dimension classified entities must be present

All vertices, all regions, all edges classified on model edgesand all faces classified on model faces

This is a sufficient minimum, not necessary but this choice allow to complete the representation without geometrical checks

Basics of the Algorithm Oriented Mesh Database

Mesh entity description A mesh entity described by a set of lower dimension entities :

Mid {M q } , d > q

All vertices are always required Vertices are atomic mesh entities, must be differentiated (e.g., using

iD (Mi0)

Mesh entities comparison Two entities are equal if their set of vertices are equal allows to compare mesh entities (<,>,=) Not absolutely general but key to practical implementation

01M

02M

11M

12M

Downward adjacencies ordering : templates

Entity described using their boundaries unique description i.e. non ambiguous shape

Weaker hypothesis Need for ordering, templates Used for computing uses T ev and T fe

Same vertices but different entities

Invert templates

Mesh Entity iD

Need of search in AOMD Add, search and remove operations are crucial Comparing entities is always possible but ... std::set<mEntity*, lessThanEntity> log behavior is not

acceptable Hash tables

Elements in a hash table not sorted complexity : worst is linear, average is constant std::hash_set<mEntity*, hashEntity, equalEntity> Hash function needed, deterministic and stateless, a mesh entity iD iD(M1) = iD(M2) M1 = M2 true iD(M1) = iD(M2) M1 = M2 false iD is a function of vertices for being independent of the representation iD(M1) = iD(M2) and M1 M2 should not happen too often, efficiency of

the hash table because equalEntity is to be used in this case

Choice if the iD

Efficiency Ne is the number of elements Nk is the number of keys

Stokes problem, tet mesh Number of dofs, use previous statistics

Counting of degrees of freedom (Szabo basis)

Higher order finite elements

Basics of the Parallel AOMD

Basics of parallel AOMD Partition boundaries treated like model boundaries

Equal order mesh entities must exist on partition boundaries (partition faces, edges and vertices)

Mesh vertices must have a unique global label On processor : serial AOMD

Implementation aspects Simplicity, no master, no owner Round of communication standardized,

no MPI calls visible, messages automatically packed

Parallel AOMD - Mesh Adaptation

Target is transient applications with thousands of mesh adaptation steps Want fast and simple adaptation Need efficient interprocessor communications

Mesh Refinement Apply templates Include support of non-conforming meshes Refined entities with remote copies must be split on all partitions Round of communication needed to ensure unique vertex ID’s

Mesh Coarsening (will be same for local mesh modifications) Collect all mesh entities involved onto one partition Carry out operation using serial operators on processor

Dynamic Load Balancing and Mesh Migration

Need dynamic load balancing after mesh adaptation Procedures build on balancing procedures in Zoltan (from Sandia) PAOMD used to provide Zoltan needed entities and connections Load balancing procedure indicates which mesh entities are to be

migrated to which processor PAOMD only migrates minimum set, unless user specifically asks to

migrate other entitiesclassification afterload balancing andbefore migration

configuration after migration

Mesh Migration

Steps in process Collect the mesh entities to be migrated to another partition Determine needed higher order mesh entities to be migrated

(use AOMD to determine minimal set needed) Collect entities and any user attached data Perform communications to send entities and update links

(following methods of the Rensselaer Partition Model (RPM)) Message passing

At PAOMD operator level it appears messages are sent one at a time This would lead to unacceptable communication costs

Message packing used - AUTOPACK (from Argonne) Automatically controls message packing process Includes information and tools to optimize message size

for network architecture used

Communications costs In the examples that follow communications were

on the order of 1% of the total costs

Implementation issues

Orientation of entities computed on the fly Language

C++ and generic programming STL, significant new feature of the language Programming with concepts

C++ and OO programming generic and OO are complementary

Trade off efficiency vs. flexibility We believe not Templates, functors… generic programming is efficient Classical example, quick sort

stl::sort is 4 times faster (with VC6) than C qsort

Parallel Autopack, automatic message packing Zoltan, dynamic load balancing and partitioning STL, associative containers, algorithms

Mesh refinement

class AOMD_RefCallback { public :

virtual int operator () (const meshEntity *) const = 0;

virtual void callback (std::list<meshEntity *> &before,

std::list<meshEntity *> &after ) const = 0; };

Conformal or not (hanging nodes or mixed meshes) Typically

class myAOMD_RefCallback : public AOMD_RefCallback; AOMD:: RefUnref(theMesh, myAOMD_RefCallback);

Communications

class AOMD_RoundOfComm { public :

virtual char * sendBuffer (const meshEntity *, int dest_proc, size_t &sizebuf) const = 0; virtual char * recvBuffer (const meshEntity *, int src_proc, size_t sizebuf) const = 0; };

Messages are packed (autopack) Typically

class myAOMD_RoundOfComm : public AOMD_RoundOfComm; AOMD::roundOfComm(theMesh, myAOMD_RoundOfComm);

Load balancing

class AOMD_LBCallback { public :

virtual char * sendBuffer (const meshEntity *, int dest_proc, size_t &sizebuf) const = 0; virtual char * recvBuffer (const meshEntity *, int src_proc, size_t sizebuf) const = 0; };

Messages are packed (autopack) Typically

class myAOMD_LBCallback : public AOMD_LBCallback; AOMD::LB(theMesh, myAOMD_LBCallback);

Demonstration of Load Balancing

Refined to moving level function - non-conforming triangular mesh

http://www.scorec.rpi.edu/DG

Desing specifications Solve any conservation law, in any dimension, in parallel

and adaptively, using any system of curvilinear coordinates, any discretization, any spatial basis and any time stepping scheme

'''''' udeurdeudeufdeudeuueee

ne

vi

e

et

gff

class ConservationLaw : fluxes, numerical fluxes fn and g, right hand side r, equation of state, initial and boundary conditions : the physicsclass Integrator : specialization's for geometrical elementsclass Metric : Euclidian, axisymmetricclass FunctionSpace : Orthogonal or not,class Solver : forward Euler, Runge-Kutta, matrix free GMRES.

Examples Navier-Stokes, Euler, Burgers, Maxwell-Boltzmann...

Demonstration of Load Balancing

2-D Examples

2D Rayleigh Taylor Red (heavy), blue (light), On left, 4 levels of refinement,

On right, 2 levels of refinement, Faster growth with more refinement

as expected

2-D Animation of Instability

Linear DG elements, 30,000 to 800,000 dof Atwood Number, A = 1/3 10 fourier modes in “random” distribution time for the bubble to reach top of the window (y = 0.5) : 5 sec

This calculation: alpha = 0.06

Experiments: alpha = 0.058 - 0.065

Theory (Glimm, et al) alpha 0.045 = 0.06

Refined 3-D Meshes for Rayleigh Taylor Instability

24steps of

refinement

72steps of

refinement

104steps of

refinement

heavyfluid

lightfluid

non-conforming hexahedron mesh

Rayleigh Taylor Instability

64 processors of the PSC alpha cluster 1 106 to 2.0107 dof’s

128 processors of Blue Horizon108 dof’s

The cannon

Conclusions

PAOMD advantages Quite small piece of software, documented Focused, mesh management only Asks for minimum user knowledge about parallel issues Efficient implementation

Future work Terascale computers, more than 1000 processors (in progress, ASCI &

SciDAC projects) Anisotropic mesh refinement (in progress, with X Li) TSTT Mesh component Hardware heterogeneity, machine and network models have to be added

in partitioners (in progress) Modification of the design, storing less