Phillip Colella

Chombo: A framework for structured-grid methods for PDE on high-performance

computers

Structured-Grid Methods For Partial Differential Equations

The original formulation of this method was for a uniform rectangular mesh.

Structured-Grid Methods For Partial Differential Equations

Mapped multiblock grids Embedded boundary methods

Block-Structured Adaptive Mesh Refinement (AMR) •  Refined regions are organized into rectangular patches.

•  Refinement in time as well as in space for time-dependent problems. •  Can be applied to both explicit and implicit discretizations of PDE.

Synchronization of Multilevel Solution

•  Average coarse-grid solution onto fine grid. •  Correct coarse cells adjacent to fine grid to maintain conservation.

Chombo: A Software Toolkit for Structured-Grid Applications

SciDAC requirement: support a wide variety of applications that use block-structured AMR by means of a common software framework. •  Mixed-language model: C++ for higher-level data structures,

Fortran for regular single-grid calculations. •  Reusable components: component design based on mapping

of mathematical abstractions to classes. •  Parallel performance: scalability to 100000ʼs of processors

with low-level details hidden from the applications developer. •  Build on public-domain standards, e.g. HDF5. •  Interoperability with other tools, e.g. PETSC. •  Not a black box: users can (and do) customize, and user

changes can migrate back into trunk. Related work: FIDIL, Titanium; BoxLib.

A Software Framework for Structured-Grid Applications

•  Layer 1: Data and operations on unions of rectangles - set calculus, rectangular array library (with interface to Fortran). Data on unions of rectangles, with SPMD parallelism implemented by distributing boxes to processors. Load balancing tools (e.g., SFC).

•  Layer 2: Tools for managing interactions between different levels of refinement in an AMR calculation - interpolation, averaging operators, coarse-fine boundary conditions.

•  Layer 3: Solver libraries - multigrid solvers on unions of rectangles, AMR hierarchies; hyperbolic solvers; AMR time stepping.

•  Layer 4: Complete parallel applications.

•  Utility Layer: Support, interoperability libraries.

A Layered approach to the Chombo design mitigates risk to the applications developers, insulating them from changes to implementations due to changes in platforms, optimizations to enhance performance.

Mechanisms for Reuse

•  Algorithmic reuse. Identify mathematical components that cut across applications. Easy example: solvers. Less easy example: Layer 2.

•  Reuse by templating data holders. Easy example: rectangular array library - array values are the template type. Less easy example: data on unions of rectangles - “rectangular array” is a template type.

•  Reuse by inheritance. Control structures (Iterative solvers, Berger-Oliger timestepping) are independent of the data, operations on that data. Use inheritance to isolate the control structure from the details of what is being controlled (interface classes).

Scaling of AMR in Chombo

Optimizations for Performance

•  Distributing patches to processors to maximize locality. Sort the patches by Morton ordering, and divide into equal-sized intervals.

•  Overlapping local copying and MPI communications in exchanging ghost-cell data (only has an impact above 4K processors).

•  Compressing the metadata that describes the distribution of boxes onto processors enabled us to scale above 10K processors.

Morton-ordered load balancing (slice through 3D grids).

Berger-Rigoutsos + recursive bisection.

Core AMR capabilities •  MHD for tokamaks using allspeed projection methods (R. Samtaney, PPPL / KAUST) •  Cosmology: compressible CFD + particles (F. Miniati, ETH) •  Space plasmas: compressible CFD + electromagnetic, kinetic effects (G. Zank, UA-Huntsville) •  Astrophysical MHD turbulence (C. McKee / R. Klein, UCB; D. Balsara, Notre Dame) •  Lattice Boltzmann method for reduction of drag using polymers (B. Alder, LLNL) •  HEDP applications (D. Lamb, U. of Chicago; exascale co-design project). Embedded Boundary Methods •  SF Bay and Delta Hydrology modelling – shallow water (CA Department of Water Resources) •  Plasma-wakefield accelerators – compressible viscous high-temperature flow (W. Leemans, LBNL) •  Blood flow in cerebral arteries – fluid / solid coupling (UCB ParLab project) •  Complex fluids at the microscale (UNC / UCD / LBNL multiscale math project; LBNL EFRC; SciDAC-e) •  Cell modeling (UCHC Virtual Cell project) Mapped Multiblock Methods •  Type II Supernovae – compressible flow (S. Woosley, UCSC; LLNL) •  4D gyrokinetic models of tokamak edge plasmas (LLNL) •  Land ice model for climate simulation – Stokesean solid (LANL) •  Atmospheric models for climate simulation – low-Mach number CFD (C. Jablonowski, U. Mich.; CSSEF proposal)

Current Chombo Applications

•  High-order accurate discretizations, including mapped and embedded boundary methods. •  AMR on mapped multiblock grids. PDE on a sphere. •  Two-level parallelism on multicore systems. Test case: AMRPoisson on 1012 cells. •  High-performance support for particles. •  Continuing work on a variety of applications: GK plasmas, LPWA, climate, multiphase subsurface flow at the pore scale, water resources, cell biology, …

Chombo – Current Development Activities

Initial Plan: Use Chombo Layer 1and Layer 2 capabilities for core AMR data structures and operations in Flash (Chris Daley, Univ. of Chicago).

Chombo in Flash

Whatʼs Next ?

Bytes per flop going down.

•  Adaptive meshes •  Scalable matrix-free methods for

sparse linear systems. •  Changing the discretizations to trade

bytes for flops (higher-order methods)

•  Models / problems with local higher compute / memory ratio - for detailed chemistry, ratio scales like (Nspecies)2 (but coefficient varies in space and time).

Grid resolution, problem complexity going up.

Applications users want same throughput on finer grids, but CFL condition on time step for explicit methods says that dt ~ (dx)p , p at least 1. More complex multiphysics problems.

Consequences: •  More implicit methods (without increased

memory footprint). •  Reformulation of problem that more

effectively parallelizes across state space.

•  Aggressive use of subcycling in time, locally in time, physical space, state space.

•  Load balancing becomes more dynamic.