The FLASH Code Parallel AMR Made Easy

Advanced Simulation and Computing (ASC) Academic Strategic Alliances Program (ASAP) Center

at The University of Chicago

The Center for Astrophysical Thermonuclear Flashes

The FLASH CodeParallel AMR Made Easy

Tomek Plewa

LACSI SymposiumSanta Fe, 10/12/2004

The ASC/Alliances Center for Astrophysical Thermonuclear FlashesThe University of Chicago

Outline

FLASH CenterAstrophysics as primary application

Verification and Validation activities

External contributors and users

FLASH CodeView from the outside

Source tree and functionalityUser level

Problem configuration

Intra-module modifications

Developer levelExtending the source tree

Inter-module communications

Summary


FLASH Application: Astrophysics

Target ApplicationsCompact accreting stars (white dwarf, neutron star)

Reactive hydrodynamics (DNS or subgrid model)

Initial conditions close to hydrostatic equilibrium (self-gravity)

Complex EOS (dense nuclear matter)

Example: Type Ia SupernovaMassive white dwarf

Subgrid model for nuclear flame

Self-gravity

Degenerate EOS


Length scales in White Dwarf Deflagration

Before 2000

Now

BG/L


FLASH Verification & Validation Program

Verification: solving equations right

Validation: solving the right equations

Verification ranging from simple analytic problems to code-code comparison.

Validation: no direct access to experiments, use scaling laws

Absolutely NO culture of validation in astrophysics!


FLASH Verification


FLASH V&V Complete Example

Experimental time series, water/glycol fog visualization of SF6 mole fraction.

Images correspond to 50, 190, 330, 470, 610, and 750 s after shock impact

Composite image does not preserve time-distance relationship


AMR-generated Timestep-dependent Signal

CFL=0.8

CFL=0.2

CFL=0.4


AMR-generated Timestep-dependent Signal


FLASH As An Open Source Research Code

FLASH is community code, freely (with minimal restrictions) available for research

External contribution: shock-cylinder experiment (Jacobs, LANL)


FLASH Center and AMR Community


FLASH Code

Primary objective

To create robust, reliable, efficient and expandable code, that stands the test of time and users.

Primary characteristics

parallel

adaptive mesh (possibly different mesh packages)

block-structured (currently)

multi-physics (hydro, MHD, SRHD, gravity, particles…)

Fortran 9x

portable

freely accessible

documented & actively maintained


FLASH Code As Seen From The Outside




















FLASH Portability: BG/L

Successful early runs up to 2048 processors

Communication is scaling well. Test size limited

by small number of available processors.

Large runs scheduled

Several days of 16K-32K processors.

Focusing on FLOPS

Make up for slower CPUs, beta-compilers


FLASH Code Access


FLASH Source Tree And Functionality


FLASH Source Tree And Functionality


FLASH Driver


FLASH Driver


FLASH Hydrodynamics Unit


FLASH EOS Unit


FLASH Unit Hierarchy

API/

Stubs

Common API Impl

API impl

Wrapper

kernel

API impl API impl

Wrapper

kernel

Wrapper

kernel


FLASH Solvers: Poisson


FLASH Source Terms


FLASH Common Fabric


FLASH Inter-Module Connectivity


FLASH Inter-Module Connectivity


Putting It All Together


FLASH Config Files


FLASH Setups


FLASH Setup Example


FLASH Initial Conditions


Configuring FLASH Application


FLASH Application Build Tree


FLASH Documentation


FLASH I/O


FLASH Visualization


FLASH Results


Summary

Overall architectureUnit typesUnit architectureNear future


Summary: Architecture

Setup toolassemble application

Config filesdefine components

Driver moduleorganize interactions

Unit ArchitectureAPI

InheritanceData management


Summary: Units Types

Physicshydrogravity

InfrastructuredrivergridI/O

Utilityprofilerruntime visualization

Data analysisFidlrflashview


Wrapper

Summary: Unit Architecture

Driver

Top level:•API

•Unit Test

Grid Data Module

patch datatime step

etc

Kernel


Near Future: FLASH3

FLASH original design was very Paramesh-likeblock structured

fixed size blocks

specified at compile time

every section of the code assumed knowledge of block size

Relaxing the constrain opens up possibilities of usingvariable block size mesh

patch based mesh

non-uniform non-adaptive mesh

uniform grid without unnecessary overheads

Documents

The FLASH Code Parallel AMR Made Easy